49

A friend of mine is working in a 200-employee company. The company's business has nothing to do with IT, but they do have an IT department to work, among others, on their website, used by the customers.

The website started with a core idea that programmers have to test the application themselves using automated testing. However, it quickly started to be problematic, as programmers were spending too much time writing functional tests with Selenium (and later Cypress.io) trying to deal with either complicated interactions, such as drag and drop or file uploads, or trying to figure out why the tests randomly fail. For a while, more than 25% of the time was spent on those tests; moreover, most programmers were pissed off by those tests, as they wanted to produce actual value, not try to figure out why the tests would randomly fail.

Two years ago, it was decided to pay a company from Bulgaria to do the functional, interface-level tests manually. Things went well, as such testing was pretty inexpensive. Overall, programmers were delivering features faster, with fewer regressions, and everyone was happy.

However, over time, programmers started to be overconfident. They would write fewer integration or even unit tests, and would sometimes mark features as done without even actually checked if they work in a browser: since testers will catch the mistakes, why bother? This creates two problems: (1) it takes more time to solve the issues when they are discovered by testers a few days ago (compared to when they are discovered within minutes by programmers themselves) and (2) the overall cost of the outsourced testers grows constantly.

Recently, the team lead tries to change this behavior by:

  • Measuring, per person, how many tickets are reopened by the testers (and sharing the results to the whole team).

  • Giving congratulation to the persons who performed the best, i.e. those who have the least tickets being reopened.

  • Spend time pair programming with those who performed the worst, trying to understand why are they so reluctant to test their code, and showing them that it's not that difficult.

  • Explaining that it's much faster to solve a problem right now, than to wait for several days until the feature gets tested.

  • Explaining that testers do system tests only, and the lack of unit tests make it difficult to pinpoint the exact location of the problem.

However, it doesn't work:

  • The metrics are not always relevant. One may work on an unclear or complex ticket which gets reopened several times by the testers because of the edge cases, and a colleague may meanwhile work on a ticket which is so straightforward that there is absolutely no chance to introduce any regression.

  • Programmers are reluctant to test code, because (1) they find it just boring, and because (2) if they don't test code, it looks like they deliver the feature faster.

  • They also don't see why fixing a problem days after developing a feature would be a problem. They understand the theory, but they don't feel it in practice. Also, they believe that even if it would take a bit longer, it's still cheaper for the company to pay inexpensive outsourced testers rather than spend programmers' time on tests. Telling them repeatedly that this is not the case has no effect.

  • As for system vs. unit testing, programmers reply that they don't spend that much time finding the exact location of a problem reported by a tester anyway (which seems to be actually true).

What else can be done to encourage programmers to stop overly rely on testers?

  • 25
    Are the programmers being more careless because there are testers or because they are under the gun to produce more code quickly? Just from your description of your problem and the way you speak about the developers, I would never work for that company. It sounds like you have managers that are not fit to deal with engineers. – John Douma Mar 26 at 21:18
  • 42
    Programmers that thing automated testing aren't providing "actual value" aren't programmers I want on my team. – corsiKa Mar 27 at 5:16
  • 9
    @BerinLoritsch Almost absolutely the opposite on metrics. Metrics can and always are gamed, and they never measure the right thing. Who had the most tickets reopened? Absolutely useless- it generally boils down to who has the most tickets, or who's tickets were most ambiguously defined. Metrics work well when the thing you measure is what you actually care about (like how long a request takes, and even then you need to understand p90 vs avg). Metrics are a HUGE NEGATIVE when they try to measure something indirectly like this. Using something like this is a sign management is clueless – Gabe Sechan Mar 27 at 17:54
  • 19
    This entire story is one management failure followed by another designed to fix the previous. If your developers cannot test the product because of complex interactions then users cannot use the product because of complex interactions. The failure of developers to write understandable, robust tests was a gift that you squandered because it was signal about vital customer-impacting issues that you ignored. Your best bet would be to obtain new management who understand the signals produced by test failures and how to react to achieve customer value. – Eric Lippert Mar 28 at 8:43
  • 7
    The specific problem of not understanding test signals is symptomatic of an even larger management problem: not understanding what testing is for in the first place. QAs job is not to test that the product works as designed. QAs job is to ensure a quality product by acting as empowered advocates for the customer to set a quality bar and ensure that it is met. Joel Spolsky once pointed out that if you make fine chocolate, you outsource the box, not the chocolate. Empowering a team to act on behalf of your customers to ensure a quality product is the chocolate; don't outsource it! – Eric Lippert Mar 28 at 14:50

15 Answers 15

63

It seems to me there is a contradiction in policy here.

On the one hand, the firm has outsourced testing because it consumed programmers' time excessively, and could be done more cheaply by others.

Now, they complain that the programmers are relying on the testers, and should be doing more testing themselves up front.

I can understand from a management point of view that there is perceived to be a happy medium, but in reality the programmers are not engaging in a close analysis, on a case-by-case basis, of how much testing they do themselves and how much they outsource.

To attempt to do so would consume too much time and intellectual effort, and likely without producing accurate results. How would a programmer go about estimating how many bugs a particular piece of code has, and then weighing up the economic benefit of spending his own time searching for them versus letting the testers search for them? It's an absurdity.

Instead programmers are following rules of thumb. Previously the rule was to test extensively. Now the rule is to save precious programmer time, get more code out the door, and leave testing to testers (who are thought to be ten-a-penny).

It's no answer to seek a happy medium, because in practice what will happen is that the anal-retentives will return to spending 25% of their time testing, and the cowboys will continue throwing low-quality code out the door, and personality traits like conscientiousness and attention to detail (or lack thereof) will predominate over the judgment. If management try to harass both types to get them to conform more closely to an average which is perceived to be economically ideal, both will probably just end up feeling harassed.

I would also remark in passing, that the 25% of time which was spent testing to begin with, does not strike me as excessive.

| improve this answer | |
  • 50
    Actually running the code to see if it works isn't a "happy medium" it's a minimum of effort expected from competent developers. Some devs apparently aren't even doing that. – DaveG Mar 26 at 20:43
  • 20
    From the OP: "they would write less integration or even unit tests, and would sometimes mark features as done without even actually checked if they work in a browser:" – DaveG Mar 26 at 21:32
  • 4
    @DaveG, I took that to be illustrative of the cavalier attitude to testing that has emerged. Otherwise it would be most surprising if the solution were that simple. – Steve Mar 26 at 22:08
  • 11
    @DaveG: Some bulky applications exist that cannot be simply spun up for a debug session. From experience, it can be a reliance on services/workers/queues, a complicated UI with no quickly accessible or transparent test data, ... (I agree these are problems, btw) I'm not saying that makes it allright, developers obviously need to to a basic inspection; but if the testing process is bulky and fidgety, and the company explicitly communicates that no time shall be wasted on testing; then the bulky and fidgety testing work is logically going to be the first that gets passed to the external testers. – Flater Mar 27 at 8:47
  • 5
    @aw04, the "happy medium" to which I refer is merely a management illusion, and I devote a significant part of my answer to explaining why it's pie in the sky. There was a working bar: the programmers spent 25% of their time on tests. Management abolished that to outsource and save money, and despite being chastised by the results, have not reverted to established patterns, and even the OP employs the language that testing is something other than "delivering features". This is not a programmer problem, it's a management attitude problem as usual. – Steve Mar 27 at 15:37
33

Bottom line: this is a cultural problem.

I come from the viewpoint that competent programmers at least write unit tests for the more complex parts of their code. The problem is not everyone shares my viewpoint. I've known people who have been developing code longer than I've been alive and they also test their code — just not necessarily with automated testing. There are a number of things in software development that are too simple to test so those tests hold no real value.

That said, there are different levels of testing with different percentages of random failures that can happen. From a management perspective, you have to understand where you gain value.

  • Unit tests: check implementation details and error handling as close to the logic as possible.
  • Integration tests: check that system specifications are working correctly
  • User Interface tests: check that the application behaves according to the requirements

Unsurprisingly, the further you get away from the individual code units, the more brittle your tests become. The more pieces you have that must work together, the more chances that something will intermittently go wrong. What that means is that the closer to the unit test you can catch problems the more reliable and valuable those tests are.

Cost of Automation

Automation costs time. Time costs money. It takes longer to write automated tests than it does to manually test a feature. However, each time that automated test is run, it runs in a fraction of the time of manual testing. From a management standpoint, you want to make sure you have a good return on your investment. I highly recommend that the highest risk code — if it breaks the application is useless — should have automated tests to ensure that you catch regressions (broken code) as soon as they occur. If the unit tests don't pass, the developer can't push their code.

In general it does help to have some guidelines, and a means of ensuring high risk code is covered.

  • Unit tests should have at least 25% coverage. (I personally prefer higher, but for a team with no unit tests this is a good place to start)
  • Unit and integration tests should be prioritized on high risk code first.
  • Definition of Done needs to have one or both requirements:
    • Code peer review (Pull requests are a great way to organize these)
    • Unit test coverage meets minimum criteria

Cost of Manual Testing

Manual testing costs time. Time costs money. While it is faster to manually test a feature one time, it takes the same amount of time to test the feature each time. You want to keep testing finished features to protect from regressions. Regressions are functionality in your application that used to work that don't any more.

The hidden cost of manual testing is that testers are people, and sometimes people skip tests on accident. If you thought writing automated tests was tedious, try testing the same features with all the button clicks time after time.

Optimizing your investment

Here's where both management and development have to be on the same page. If quality is important to the company, then the company has to be willing to invest in quality. If quality is not important, then just do away with testing. Your users will complain, and you may or may not be embarrassed by the problems they complain about. That said, if the application is mission critical, then quality should be important.

  • Automate testing high risk code
    • Risk can be high due to the complexity of the solution
    • Risk can be high due to how necessary the feature is
    • Risk can be high due to the high number of dependencies on the code
  • Don't write tests for code that is too simple to fail (like getters and setters)
  • Manually test things that are too complicated to test automatically (like drag/drop)
  • Invest in simplicity
    • A requirement on its own might be simple enough, but may conflict with other requirements.
    • Be ready to remove features so the application serves current needs.
  • Define "done" so it is clear and unambiguous
    • When the work that needs to be done is unclear, then developers and testers have different opinions on what's right.
    • A few more minutes in a meeting with three people can save days of work and rework because of differing definitions of done.

Summary

The company culture is currently in a no-win situation. Culture changes are easier when management has buy-in. It's also easier when the team introduces disciplines that help them be more effective. To that end, I would prioritize Defining Done before prioritizing anything to do with how tests are performed.

It's great that you are collecting metrics. It's not great how those metrics are currently being used. A better way is to look at the trends on the metrics as you introduce more structure in how your team develops software. For example, if time to completion is improved and number of test failures are down because you spend more time defining what needs to be done, then that is a win. If you increase your automated test coverage to 50% and don't see any improvement in the number of test failures then maybe 25% is good enough.

Software development is a team activity. The more you work together as a team, the better everyone's attitude will become. The more your team is set up for success, the more your team will experience success.

| improve this answer | |
  • Not sure if this is covered by Risk can be high due to how necessary the feature is, but i'd also add that risk can be high due to an elevated number of users that depend on the feature, or an elevated number of processes that are affected by/related to the feature – Josh Part Mar 28 at 16:10
  • That's what the second bullet there says. I.e. how necessary the feature is. I will add another bullet for the integration piece though. – Berin Loritsch Mar 28 at 17:11
  • Great answer, but I think it should be expanded by pointing out the fact that a culture is company wide. You can't change culture in your department only. In this case I have the impression the dev team is seen as feature machine and not enough time is spent with the developer and stakeholder to discover the problem and in that way gain a shared sense of responsibility. – winkbrace Apr 3 at 7:46
16

Okay, we all share the common goal of protecting and defending developers but there are some things in your question that make me somewhat uncomfortable...

They also don't see why fixing a problem days after developing a feature would be a problem. They understand the theory, but they don't feel it in practice.

I am sorry to break the news, but experienced programmers do feel the value of catching a bug early in practice.

As for system vs. unit testing, programmers reply that they don't spend that much time finding the exact location of a problem reported by a tester anyway (which seems to be actually true).

This is textbook defensive talk (smells somewhat passive-aggressive). And of course it's true....until you put it in perspective. You can ask the programmers... do they use bookmarks in their browsers? Do they home-page a search engine? Do they use alt+tab to switch between applications/windows? All these things are pitiful excuses of productivity gains... that is if you do them ONCE. When you do it all the time, those few seconds gained easily sum up to countless man-days of productivity. It takes your programmers what, 1 minute to find the source of a problem reported by the tester? That's the time they felt. The actual time includes the time of the tester, the time to prepare the report/file the ticket, the time potentially spent in communicating the details and the time to close the ticket/report it as solved. And time is money, right?

Think of this like forgetting to buy milk when you go buy food. While you're IN the store, it's 1 minute of walking. When you are at home, it's exponentially more. Now, to put it in perspective, this is how 1 minute of felt time translates to ~30 minutes of actual trouble for all involved (though it is probably far more in reality). If management knew that proper testing could save the company some 30x the time occasionally spent given the current situation, how would they feel?

Furthermore...

For a while, more than 25% of the time was spent on those tests; moreover, most programmers were pissed off by those tests, as they wanted to produce actual value, not try to figure out why the test would randomly fail.

Again, I am sorry to break the news, but experienced programmers understand that there is actual value in tests plus 25% of the time is actually not that bad a percentage. Good tests are worth their length in gold! Also, tests don't randomly fail, at least not as often as this bold statement would have you think. Not good tests, at least, so the quality of the tests is also one thing that will have to be seriously considered. Think about the CE mark, for example, which is all about testing! Do the programmers equally disagree that it does add actual value to products?

The actual trouble with this last quoted OP statement, however, is that it seems to hint that management has actually taken advice from the programmers to outsource testing, or at least been affected by the programmers' aversion to testing, unlike how other answers may suggest that management screwed this up all by itself.

Programmers are reluctant to test code, because (1) they find it just boring, and because (2) if they don't test code, it looks like they deliver the feature faster.

Well, management does screw up occasionally, right? It seems that pressure has been built on delivering features quickly, and this has pervaded the culture a bit. Cutting corners and dropping tests go hand-in-hand when under excessive pressure to deliver, unfortunately.

Experienced programmers would go a long way over how tests are worth the time, rather than complaining that they eat up their time and they don't get to deliver value, as they say. I am in the devil's advocate position of having to give the simple explanation that the programmers are relatively inexperienced, but also, the entire system they are juggling with is not that complex, so they have not had too much trouble yet, to really feel the value of proper automated tests.

There is only one suggestion here I would be in position to give and this is to strive to make the developers feel the actual value of tests at all costs. Also, while at it, keep the external testers and compose proper automated tests to the extent possible. If the output is not too complicated, safety-critical, etc... the company could probably do without the external testers, but that always depends on the details.

| improve this answer | |
  • 1
    Some excellent points here. However, that last paragraph feels totally out-of-place, and does not add any value to an otherwise excellent answer, imho. – cmaster - reinstate monica Mar 27 at 7:51
  • 2
    yes the condom analogy seems a bit awkward out of place. Other than that, one correction " good tests don't randomly fail on good software". For example, if you don't consider concurrency in your tests and your application you may end up with tests randomly failing if you run multiple tests in parallel for instance. That would however be a sign that either you need to redesign your tests, how you run them or even your software because it has a concurrency bug. – Frank Hopkins Mar 27 at 10:16
  • 3
    It's fair to say the author should have stopped sooner. – Steve Mar 27 at 15:42
  • 1
    It's fair to acknowledge a failed attempt at humor! :-X I am removing the last paragraph and integrating the correction suggested by Frank! Indeed, tests also have to be good. – Vector Zita Mar 27 at 17:52
6

Don't just assume you're right about over reliance on integration testing. You're pushing for the wrong things. Be very careful about driving people to measurable results vs actual results.

Measuring, per person, how many tickets are reopened by the testers (and sharing the results to the whole team).

I’m sorry but this is just dumb. The programmers job isn't done when code is in the hands of a tester. Counting how many times code goes back and forth between them against the programer is counter productive. You want programmers engaged with testers. Don't punish them for doing it. Good programmers cheer on their testers. They don't hide from them.

Giving congratulation to the persons who performed the best, i.e. those who have the least tickets being reopened.

This is not a good measurement of best. You're rewarding those who can hide their bugs, not those who work to see them eliminated.

Spend time pair programming with those who performed the worst, trying to understand why are they so reluctant to test their code, and showing them that it's not that difficult.

Everyone who runs their code is testing their code. When people don't understand how they're being judged the smart ones don't even try to guess. They just quickly submit to the judgement and hope to learn something useful from the result. We have a fancy term for this behavior. It's called agile.

Now please don't take that to mean I don't like pair programming. I love it. I've learned more sitting with fellow coders banging away on a problem then from any book or blog. Can't wait for this covid thing to go away so I can get back it doing it.

But it's not pair programming if all you do is preach. If you do this, shut up and listen to your partner. You may learn something and find your problem isn't what you think it is.

Explaining that it's much faster to solve a problem right now, than to wait for several days until the feature gets tested.

Well here's your problem. It shouldn't take several days to test a feature. Think of code like cement. The longer it sits, the harder it is to move. Get me tester feedback faster. Preferably before I start thinking about other features.

The longer it takes to catch a bug the more expensive it is to fix.

Explaining that testers do system tests only, and the lack of unit tests make it difficult to pinpoint the exact location of the problem.

Stop explaining and do peer unit testing. It shows them what you want. Programmers communicate best with code. This lets them do that.

Works like this: I write a unit test. I write production code that passes my unit test. You peer review the code, notice something hinky, and you write a unit test that proves hinkyness. You send your unit test to me, I look it over and either make it pass or we talk about what the real requirement should be. Do peer unit testing and your weak unit testers will learn from your stronger ones. Strangely enough, the strong ones will also learn from the weak ones.

Oh and you can do this when you pair program. Makes the whole thing go fast.

Speaking of requirements, if the integration tester isn't part of the discussion about the requirements, and which ones are testable, before this coding started you're out of your mind. They make the call on which requirements are integration testable. Coders should know how this shook out before they even start on a design.

You send code to integration testers after it's been through peer review & peer testing. Now two coders have picked this over early and both of them should be rooting for the testers to find any remaining bugs. They shouldn't be trying to hide them. They should be openly reporting anything that might help the integration tester find them.

It's not programmers vs testers. It's all of us against the bugs.

So stop rewarding people for creating untestable code.

| improve this answer | |
4

I'm firmly in the 100% automated coverage camp. I think it actually helps me go faster in the long term, and I hate having things kicked back to me from a tester.

That being said, if people hate doing automated tests, you need to provide an incentive to not have to write them. If they think they can still maintain quality without automated tests, let them prove it. I would require automated tests for any bugs that are kicked back to them from testers, and make them optional otherwise. In other words, they need to write a test that fails if the reported bug is present, then show that it passes with their fix, as a prerequisite to getting it retested by the testers.

What I hope would happen, is the sloppiest programmers will end up having to write the most tests, which should encourage them to take more care. Code that is the easiest to break will end up with the best coverage. People will likely start preemptively writing tests for things that are difficult to test manually, and doing more thorough manual tests for tests that are more difficult to automate. Hopefully there will also be some investment in making tests easier to write and run. Very little will change for programmers who are already careful.

| improve this answer | |
3

Consider taking this up in the Project Management thread.

But also – simply tell them that their work will not be accepted unless it is accompanied by automated tests. "Yes, this is part of your job." The external testers are there only to keep you honest.

As a lifetime software developer myself (as well as a consulting project manager), I can directly speak to the number of times when I feel that "my a*s was saved" by the tests that I wrote, even of my own code that no one else would work on, as I was building it. I frequently "failed to get it right." And I was sometimes astonished when an earlier test suddenly started failing. I found-and-fixed one "gotcha!" after another ... before it had a chance. And in this way I built layer upon layer of code knowing that it was correct.

Get management support behind you that "it's not accepted until it's tested, and we reserve the right to reject your tests."

| improve this answer | |
  • 9
    It seems to me you're failing to appreciate the likely role of management in producing the problem. Originally it was said that programmers were grumbling, but not that they were failing to test to a high standard. The overall theme is a familiar one: management try to cut costs by cutting corners, then quality falls. – Steve Mar 26 at 20:47
3

A couple of points I'm not really seeing in the other answers.

The skill sets for development and testing may overlap but some people make great testers, and sometimes they're not the same people that make great developers. As a developer, a good QA department is worth its weight in gold; forcing me to do all the tests myself, especially in an environment where someone else's code might change the behaviour of a web page between when I test it and when the combined set of changes goes live is ... the opposite. So, if you have a constant stream of changes, testing the finished result after it's been merged is quite a bit more important.

A developer will likely be able to tell you what needs testing, what could go wrong, and so on. Listen to them.

Another point I'm seeing in the OP is that at some point, after your developers had got used to using one testing framework, there was a decision to switch to another, which boasts on its website of reducing the volume of integration tests, and so on.

You didn't mention the reasons this decision was made, nor whether any training or learning time was set aside for this change. Were they expected to learn this new framework on top of continuing to work at the expected pace on feature development, bug fixing, and so on?

| improve this answer | |
  • I agree with this. Testing is a specialism. In the simple cases I could be both bricklayer and dynamite man to an adequate degree, but there comes a point where the person who knows how best to put it up is not the one who knows how best to bring it down. – Steve Mar 28 at 13:09
2

Reviews may work better in that environment.

As a developer I feel the same. Writing tests is dead boring (particularly unit tests). And they are not always effective (particularly unit tests). It depends on the type of logic you write and on the kind of input you get. 100% specified requirements are not that common in my experience so what would your tests prove, really? Requiring people to write unit tests just because can be demotivating. So can shaming people who produce the most "bugs" (they may well be the same ones producing the most value).

Peer pressure is more effective. You have to get used to one another and this may hurt a bit but in the end you will have a team. You will not get a team by imposing external rules that are not supported by the people that have to follow them.

I would even suggest to ditch the off-shore test guys altogether. I mean, why would I try hard to deliver flawless behavior when I am not trusted to do so anyway and there is another layer of verification going over my work anyway? Apparently this is the way we work, right? While they are there, let them make my life a little easier and find the loose ends for me to fix, so I don't have to put too much effort in it.

If that safety net weren't there... I would be more careful. I would feel respected and responsible. With an attitude like "you are going to deliver crap so we will have everything you produce scrutinized", I would not feel inclined to try very hard.

| improve this answer | |
  • 1
    I personally think that effective testing depends on a significant body of specialist skill and knowledge, and the inclination to be involved with it differs between individuals (just as with any sort of skill or area of knowledge). The architect who designs buildings does not necessarily want to be the person who builds the scale models and builds individual wind tunnels to test them - even if he would be content to turn his hand to any of these tasks, to turn his hand to them all in one project involves too much complexity for a single mind. – Steve Mar 26 at 21:13
  • 3
    I know that I am a good developer, but I really suck at testing, especially exploratory testing. I love delivering high quality software and I actually feel good if a good tester finds it hard to discover a flaw in my software. For me, it is a good thing that my software gets additional scrutiny by a test team before it is released into the wild. It might also matter that I work in industries where a bug in the wild costs the company real money and goodwill as products may have to be recalled for repairs. – Bart van Ingen Schenau Mar 27 at 7:52
  • 1
    Just to clarify: You're saying testing is boring for you, as a developer, and you're also recommending getting rid of QA. So that means you're recommending no testing whatsoever and you trust there will be high quality because developers will just try harder (and review well)? – Bernhard Barker Mar 27 at 23:02
  • 1
    Testing (particularly unit testing) is not supposed to always be effective at catching bugs, because there may not always be bugs to catch, testing well / writing good tests is hard and testing absolutely every possible thing that can happen is just not realistic. It's supposed to catch bugs sometimes, provide peace of mind that your code is working as intended, make it easier track down bugs earlier and become a habit so you do it when needed. Also, you shouldn't need external requirements to inform tests (although it helps). If you know what the code does, you should know what to test. – Bernhard Barker Mar 28 at 0:21
  • 1
    @Dukeling, when you say "If you know what the code does, you should know what to test", I think that's precisely the problem. For many, the purpose of testing is because they aren't sure they know what the code does - if they did, testing would be needless in their circumstances. In larger, longer-lived codebases, tests are not altogether about finding current bugs, but mitigating the risk of future bugs that arise from unintended breaking changes maybe 10 or 20 years down the line, or mitigating the risk of bugs from large teams who don't all fully understand the whole. – Steve Mar 28 at 12:57
2

Our organization solved the problem of developers never actually testing their changes themselves by making that part of the process:

  1. Giving a change to testing requires that you provide documentation of your changes.
  2. Documentation must be made using the standardized template.
  3. That template includes the following mandatory sections:
    • What was the reason for the change?
    • What was changed?
    • A "Functionality Test" where the developer does a basic test of the change, documented its execution and results with screenshots in a way which shows that the goal of the change was achieved. This test is not expected to be as thorough as a test performed by the actual QA team. It is just supposed to demonstrate that it now appears to work as far as the developer is concerned.

That way every developer is forced to run their changed code at least once.

| improve this answer | |
1

You need a "definition of done"

Generally accepted 'definitions of done' include not only working code, but working unit tests, and documentation. Until those things are completed, the task is not done and the code cannot and should not be released to the testing department or god forbid, live.

While you'll often find the concept under the Agile/Scrum umbrella, that working framework isn't necessary to implement this idea.

https://www.scruminc.com/definition-of-done/

https://www.agilealliance.org/glossary/definition-of-done/

The testers are there to catch the things you didn't think of, weird use cases, or things honestly missed.

| improve this answer | |
  • Plus, I think, a review of the code and tests by another developer. – Robin Bennett Mar 27 at 17:07
1

You can encourage developers not to "overly rely" on testers by focusing on the results you're really trying to achieve (presumably features that balance the cost of development with the value the company gets from them being deployed at a certain quality level) and letting the developers decide how much to rely on the testers in order to achieve this goal. In other words, unless you're part of the development team, you're trying to manage something that the development team should be managing.

Fixing "Done"

The first thing you need to do is fix your definition of "done." You say that developers,

...would sometimes mark features as done without even actually [checking] if they work in a browser: since testers will catch the mistakes, why bother?

First, redefine "done" as, "the change is ready to go out into production, for use by real customers." Second, make each developer responsible for management of the process that gets any particular change to the "done" stage. That includes ensuring that whatever testing the developer considers necessary is completed, whether that be by writing and running automated tests, sending code and test instructions to human testers and reviewing the results, or anything else they can think of.

If a change is found to be defective in some way after the developer has declared it "done," that's the time to sit down with the developer and do some review. This review should be between the developer and one or more senior members of the development team. (It might or might not also include someone from the product owner side; see below.) This review should not be confrontational and certainly should not start out by assuming something was done wrongly, but examine the impact of the released (or about to be released, if you caught it in time) defect and consider what cost-effective measures could have been taken to avoid that happening, i.e., what the developer (and the development team) should change to improve things. (The answer may well be, "Nothing. The cost of preventing all or most things like this is more than the cost living with them.")

The product owner in such meetings, if present, is not there to push the team for technical or procedural changes but for two other things: 1. to provide information about the cost of the negative business impact from the release (or estimated cost, if the defect got caught in time), and 2. to evaluate whether or not the product owner wants to pay additional costs to prevent this kind of problem. (E.g., the developers say, "we can reduce the incidence of this kind of thing by 50% but it will slow development by 5%"; the product owner can then decide whether the increased quality at release is worth the slower rate of getting features rolled out.)

Developers Manage Testing

Sending changes out to human testers is, at a high level, not much different from writing automated tests. There's some sort of script for the testers or test system to follow (even if it's a simple as, "go to this URL and tell me what you think") and the developer evaluates the results to decide whether she's done or she needs to do more work (be that changing code or running more tests).

This script needs to be kept somewhere where it can be reviewed by other developers at (or in preparation for) the meetings I described above. And the history of changes to it needs to be kept as well, so that e.g. if a developer removed something from it (perhaps because she added automated tests to cover something that was problematic or taking too long for the human testers to do) the review can examine that decision. (Personally, I would usually keep this information in the source code repo itself, in text or markdown files, but the developers should do whatever works well for them and the test team.)

The initial script would probably include the story from the product owner, instructions about how to set up anything needed for testing that bit of code, and any other notes the developer finds useful to throw in.

As with anything, developers not experienced with this will probably need some support from others in the development team. You wouldn't expect a developer with little experience in unit testing to be able to write good unit tests right off the bat, or even figure out what needs to be tested; the same goes for managing manual testing.

But the developers need to keep their eye on the goal here: to be able to declare something as "done" with the meaning that they think it will pass all acceptance tests done by the product owner, or that would be done by the product owner if he didn't trust the developers so much, and even any tests that the product owner might think of after the developer's got the story and started coding it.

Acceptance

There's something I somewhat elided above, which is that there really are two "done" stages, under the control of separate teams. The first is the development team's "done" stage, which means "I expect that this will pass all acceptance tests," as mentioned above. The second is the product owner's "I accept this change; let's release it." How much work the product owner does on this side depends on how good a job the development team is doing. Where a development team has proven themselves very reliable at interpreting the requirements and writing code that meets them, the product owner may do minimal testing. Where the development team is not doing such a great job at this, the product owner will have to more verification.

Where you currently stand on this spectrum is not too important: what is important is that, when acceptance testing fails, the development team reviews how it failed (i.e., why they incorrectly felt is was "done" when it wasn't) and develops systems to prevent future similar failures at acceptance testing. Improving this in any particular instance may be a purely internal technical matter for the development team to deal with, or some failures may be due to communications problems between the product owner and the development team, in which case they need to work together to figure out how to fix that.

Summary

Writing automated tests and sending code off to a test team are simply two different methods of acheiving the goal of ensuring a change is of sufficient quality to release. They're almost invariably used together, and the decision about how much of each to use needs to fall to the developer making the change, because it's intimately affected by how she chooses to write her code. So she should be manging that, and needs whatever support is necessary to get good at managing that.

| improve this answer | |
0

It seems that the policy went from one extreme to the other. I've never seen working in practice a test coverage of 100%. The trade off between the amount of time and detail required to write a test for a function and the benefit of the test itself is usually very variable. Mandating in a blunt manner a 100% test coverage without considering the trade off will eventually force the developers to take more time than needed to write some tests.

A flexible approach would be better. Require a minimum of test coverage and tell the developers to try and judge how difficult would be to write the test for a function. However another person who is not the developer himself should do some review, check which functions are subject to frequent changes or have such complex requirements that they could be broken easily and flag those functions as the ones that need a test no matter what.

Another thing that would help is finding the developer most skilled with the testing tools and ask him to develop some templates for the most difficult part. Often finding how to build the mocks around a particular function really takes a lot of time. A set of patterns/solution close at hand would save a lot of time.

| improve this answer | |
0

There's a very simple solution.

You call a meeting with your developers. In that meeting you tell them: "We have a problem with our process right now. It seems that at the moment, the code that is submitted very often does not work properly. That causes QA to spend too much time writing bug reports, and then costs development time fixing those bugs. We had the problem previously that development was extremely high quality, but at huge cost. We need to get more to a point in the middle, where the developers spend a bit more time to make sure there are not too many problems passed to QA, so we can work more efficiently overall".

Some answers are too focussed on the difficulty to find the best point. That's not required. You had a point at one extreme which was inefficient, you have a point at the other extreme now which is inefficient, going further to the middle will be an improvement. Don't ignore an improvement just because you cannot find the optimal point.

| improve this answer | |
0

You have just answered for you questions, when have the two topics published: "Programmers are reluctant to test code, because (1) they find it just boring, and because (2) if they don't test code, it looks like they deliver the feature faster." What does it mean? How the delivery is done, if there are no tests, which are planned. Any project/module can be delivered (even put to a GIT) repository only after maven or gradle or what else have executed all the tests along the build. What is an input of development of a unit? Analysts, or architect must provide a list of cases, which must be subject of tests for a component, or in worst case, testing engineers (are responsible for tests) first put tests into repository and developers get them upon pull request. Maven/gradle will do the work. Besides a delivery must pass a code review with first attention to test. And only then- changes can be merged with branch which us used for industrial build. No tests- no delivery, work is not done! That's all.

"Spend time pair programming with those who performed the worst, trying to understand why are they so reluctant to test their code, and showing them that it's not that difficult." Really, XP's element as Pair Programming is not dedicated to display ABC to developers, which are reluctant to do a part of work. Testing - is not only testing, it is a part (an important) of their work, that must be in the definition of position. The approach you have mentioned- is spending time not only of testers, but of another developers-unless it is on the first stage of a newcomer (beginner!) adsorption. You environment, including DevOps must be built in accordance to the principle: only a component, which passes all the PREDEFINED tests is to go. Otherwise- it is not delivered.

| improve this answer | |
-2

You'll need to go back to developers doing testing.

Where I work, every change pushed out by our team is tested by two other people on the team. The originating developer provides a test plan, and the two peer testers run the test, any other tests they feel like running, and do code review as well.

If it passes this, then our QA department gets a hold of it.

The only drawback I've encountered with this strategy is that some of our developers would be doomed if their lives ever depend on their ability to write clearly.

| improve this answer | |

Not the answer you're looking for? Browse other questions tagged or ask your own question.