23

I had a discussion with a testing manager about the role of unit and integration testing. She requested that developers report what they have unit and integration tested and how. My perspective is that unit and integration testing are part of the development process, not the testing process. Beyond semantics what I mean is that unit and integration tests should not be included in the testing reports and systems testers should not be concerned about them. My reasoning is based on two things.

  1. Unit and integration tests are planned and performed against an interface and a contract, always. Regardless of whether you use formalized contracts you still test what e.g. a method is supposed to do, i.e. a contract.

    In integration testing you test the interface between two distinct modules. The interface and the contract determine when the test passes. But you always test a limited part of the whole system. Systems testing on the other hand is planned and performed against the system specifications. The spec determines when the test passes.

  2. I don't see any value in communicating the breadth and depth of unit and integration tests to the (systems) tester. Suppose I write a report that lists what kind of unit tests are performed on a particular business layer class. What is he/she supposed to take away from that?

    Judging what should and shouldn't be tested from that is a false conclusion because the system may still not function the way the specs require even though all unit and integration tests pass.

This might seem like useless academic discussion but if you work in a strictly formal environment as I do, it's actually important in determining how we do things. Anyway, am I totally wrong?

Improve this question
19
  • 10
    Does it matter?
    – yannis
    Jun 5, 2012 at 15:42
  • 5
    @YannisRizos From the title, no. From the entire question, it certanly seems so fro the person asking
    – Ludwig Magnusson
    Jun 5, 2012 at 15:47
  • 2
    @Rubio From your question I agree that reports on unit tests are useless for the system tester. Unit tests are a helpful tool for the developer. How does your testing manager motivate the need for these reports?
    – Ludwig Magnusson
    Jun 5, 2012 at 15:49
  • 2
    @LudwigMagnusson True, however if it only matters to the person asking, that's too localized.
    – yannis
    Jun 5, 2012 at 15:56
  • 6
    developers reporting to testing manager is wrong, no matter what. If she passed that request through your (dev manager), you'd just have to do what your boss told you. "Talk to my manager" is the weapon one need to know how to use in "strictly formal environment" as yours
    – gnat
    Jun 5, 2012 at 20:01

10 Answers 10

Reset to default
30

Writing automated tests is a developer's job; the tests are part of the codebase and should be treated as such - they should be subject to the same code reviews, coding standards, source control discipline, etc., as the rest of the project.

Running said tests is done for two reasons: First, as a tool in guiding the developers. You run tests to verify that the code you have just written does what it is supposed to, you use them as additional documentation, and to verify that changes do not break any existing functionality. If you do real TDD, the tests are also authoritative source of technical specifications. The second reason to use the tests is during QA and deployment. Running all automated tests should be one of the first steps in every round of testing; running automated tests is cheap (virtually no manpower required at all), and it doesn't make much sense to go into manual testing if the automated ones fail.

This means that the responsibilities should be like this:

  • Developers write automated tests
  • Developers run individual automated tests as needed, as part of their development workflow
  • QA runs all automated tests as one of the first stages of testing

If you have a build server, then QA's task (regarding automated tests) boils down to "open the build server's report and verify that everything is green".

Improve this answer
8
  • Excellent post, exactly along the lines I was going to post. Only qualm is relying too heavily on unit tests as well as integration tests can lead to issues down the road. I disagree with the fact that QA's task boils down to checking the build server (I'm assuming you mean something like Hudson). This is putting all of the testing burden on developers to write tests that cover ALL business logic, all the time, which seems to be putting too much weight on the developers.
    – dardo
    Jun 5, 2012 at 19:44
  • 4
    @dardo: Of course automated tests aren't the only tests you should run, otherwise you could just get rid of QA altogether. That would be ridiculous - a lot of very crucial aspects of any software product simply cannot be tested automatically. What I mean is that given the existence of automated tests, QA shouldn't have to worry about them beyond checking the build server's output; after that, they do their normal thing - manual and semi-automated testing on the completed build.
    – tdammers
    Jun 5, 2012 at 19:50
  • Ah yes, agree 100% Sort of like a checkpoint, are they there, do they pass, etc.
    – dardo
    Jun 5, 2012 at 19:55
  • @tdammers - Testing is only a small part of quality assurance.
    – Matthew Flynn
    Jun 5, 2012 at 22:10
  • 2
    Excellent answer, however I don't agree that tests should be seen as an authoritative source of technical specifications. Tests should be a confirmation of specifications, but certainly not a replacement. That is not to say that I am advocating for a big up front spec either, but rather that I'm making the distinction that we apply tests to validate the things we know about the way in which a system should behave, rather than having the tests define the behaviour. A pedantic distinction perhaps, but important nonetheless.
    – S.Robins
    Jun 5, 2012 at 23:52
10

I think the most important for you would be to clarify why she needs that report.

There can be different explanations (as suggested by several answers), which require very different strategies.

  • if she is a reasonable person, simply wanting to get information to help the work of her testing team, it makes sense to get to a common understanding, and work out some solution which would be suitable to both of you. You can discuss with her the nature of unit tests and the fundamental difference between unit vs functional / system / acceptance tests. Hopefully you can get her understand that these work on very different levels and neither can replace the other.
  • if she is a control freak or bureaucrat, demanding a report just for the sake of it, you can generate something to satisfy her whims with the least amount of effort (e.g. what @Doc suggested :-).
  • if she is into some power game, you may question whether she has the right to demand reports from developers. In my experience, developers usually aren't supposed to report to the QA department.
Improve this answer
2
  • Good points. She does seem like a reasonable person. My fear, which I didn't make very clear, is that the unit tests lead the testers in the wrong direction and false security in what they need and need not test.
    – Rubio
    Jun 5, 2012 at 16:49
  • 2
    @Rubio, indeed, you should clarify to her that unit tests can't replace system tests. In fact, high unit test coverage of a specific module may even be a warning sign that that module needs extra attention during system testing! If the developers took the extra pain to write that many tests, the code may have been drastically changed/extended lately, and/or is full of bugs.
    – Péter Török
    Jun 5, 2012 at 20:59
7

I think the role of QA and Development, and the interplay, can vary a lot between organizations. But in general, on my team, I tell joining members to basically pretend like there isn't a QA team, in the sense that they are responsible for the changes they are pushing into production. In turn, our QA team doesn't assume much about developer testing, and does a fair amount of testing the system as a functional whole.

For this reason, our QA team doesn't care all that much about what is and isn't unit tested before they begin testing.

I do think it's helpful for the QA team to understand what the unit tests do and do not cover, at a high level, so that we can collectively work to identify gaps, and areas that might need more rigor. So, maybe your colleague is after a high level summary, as opposed to the gory details.

Improve this answer
5

She insisted that devs report what they have unit and integration tested and how.

Is she really trying to argue about whether this kind of testing is actually in the realm of "development", or is she just trying to figure out how well your code is covered by unit testing? Just by looking at the information you've given, it seems like she just wants to know which parts of the code are covered and where she should focus her team's effort.

I worked on a testing team right out of school before I moved into a development role, and I can see how this might be valuable to her and her team.

Improve this answer
4
  • 1
    But shouldn't the focus come from the specs? There are situations when code changes have unexpected repercussions and then it's very important for the dev to communicate that testing should also cover this and this.
    – Rubio
    Jun 5, 2012 at 16:10
  • 1
    @Rubio: Of course, but from a purely practical point of view, try and look at it from her perspective. Assuming that all parts of the application are not going to have perfectly equal amounts of code covered by the unit tests, wouldn't you want to dedicate more of your limited resources to the parts of the application with less coverage? To me, it's simply a matter of looking at the report and saying to my team, "Hey guys, it seems like Area X has less code covered by tests than Area Y, let's try to focus on running tests on Area X"
    – Jason
    Jun 5, 2012 at 16:58
  • @Rubio: yes, but if you're doing TDD (ie BDD) properly then your tests should be against the specs in the first place. If your company was truly enlightened, then the test team could write the tests for the dev team.
    – gbjbaanb
    Jun 5, 2012 at 18:28
  • 2
    What is tested: automatically-generated code coverage report. How is it tested: read the unit test code. @gbjbaanb: "the test team could write the tests for the dev team." There are so many things wrong with that statement, that I don't know where to begin, except to say, Very Bad Idea.
    – BryanH
    Jun 12, 2012 at 21:12
5

I don't see that it matters too much.

If you don't provide them to QA/Testing, and they don't do proper tests, and it fails in production, it's their fault for letting it through QA into production without verifying it works as specified.

If you do provide them to QA/Testing, and they don't do proper tests...same outcome as if you had not provided them.

However, if you provide them, they could compare them against the spec as well, and/or suggest which tests might be flawed, or need changing because they did find a bug.

Really, I don't see much downside in providing them. It's still on QA/testing to validate against the spec. If they take the lazy way and just rely that your testing is good enough because they all passed, it's them that failed at their job. As long as they still have the spec as well, the results of the unit/integration tests are just fluff, and shouldn't be able to hurt you one way or the other. This is the reason we have dev and QA. Multiple checks that the app performs as specified.

Devs make mistakes, QA makes mistakes, ideally they don't both make a mistake on the same item...and if they do...it's potentially a analyst who dropped the ball writing an unclear spec.

Improve this answer
5
  • 2
    The downside to me is extra work that provides no or little value.
    – Rubio
    Jun 5, 2012 at 16:04
  • @Rubio, what extra work? Just print out the result. If they are named well, it tells them what you are testing. The shouldn't need the actual code or description of how the method works. If they do, they can look it up themselves.
    – CaffGeek
    Jun 5, 2012 at 17:34
  • 1
    Generating a report of the 3500 unit/integration tests that passed would probably be of little help to tester, even if the tests were well named (which they should be but aren't). In order to provide testers with meaningful information about unit test, the dev would have to dig through the unit test associated with a particular feature and somehow communicate to the tester what is actually tested and how it's asserted. That's a lot of work.
    – Rubio
    Jun 6, 2012 at 14:50
  • 2
    @Rubio - automation is your friend. You can set up a continuous integration server that mails reports any time there is a checkin (this will help you too). If QA is requesting explanation of tests and code, then it sounds like they've gone beyond the level of reasonableness and into the realm of "failing to grasp the concept". If they can't or won't read the code, then they are useless. At that point, a chat with your manager would be beneficial, and you can lay it out like, "QA wants me to spend x% of my time helping them read code, is that okay?"
    – BryanH
    Jun 12, 2012 at 21:09
  • 1
    +1 for saying that it doesn't absolve QA of their responsibility to independently test the software.
    – user4051
    Jun 12, 2012 at 22:26
2

Unit testing is the responsibility of developers that the tests can be useful to understand how the pieces of code work on their own. Some may see this as a form of documentation and thus has some value although there may be overhead if the unit tests are changed regularly.

The other value in passing the tests over is that this can avoid doubling up on tests that may be redundant in terms of ensuring basic functionality.

There is also user acceptance testing that is separate from all of this as the end user may have their own understanding of how a system is to function.

Improve this answer
1
  • 1
    Redundant testing is what's often used as an argument and may sometimes be true. However, system testing is always performed on the entire system whereas unit/integration testing focuses on a specific unit. I see a danger here.
    – Rubio
    Jun 5, 2012 at 16:06
2

If your company has a defined methodology to ensure the quality of its products (if they are SOX compliant, or are trying to improve their CMMI level, they probably do), then products must be able to stand up to audit to show that the process was followed.

Often, the defined process includes unit testing (which is a good thing). Unfortunately, this also means you have to document your unit tests and prove that they were run in order to stand up to audit. So that means you need a way to report on your unit tests.

Look at a tool like Sonar to help you out--it will report the level of code coverage and the results of your unit test runs.

Improve this answer
3
  • SOX no, CMMI yes. Our unit/integration tests are part of the code review process and that does stand up to audit. I can get a generated report from a unit/integration test run, but that's pretty cryptic for a tester. Coverage is also in the report but that in itself means nothing.
    – Rubio
    Jun 5, 2012 at 16:17
  • First, don't give your tests cryptic names. Check out dannorth.net/introducing-bdd. Second, code coverage may not say much about the quality of the tests, but it at least shows that the units being tested do no blow up when most to all of the code is run.
    – Matthew Flynn
    Jun 5, 2012 at 19:39
  • Good link, thanks. I remember reading an excellent magazine article exploring different ways to name unit tests but can't for the death of me find it now. May have been Visual Studio Magazine or Code Magazine.
    – Rubio
    Jun 6, 2012 at 14:55
2

This really depends on the company but from my experience having worked both as a system tester in a traditional approach but also as a tester working in an agile team in a CD model, knowing what has been unit and integration tested is extremely useful.

Quality is the responsibility of the team - both developers, testers and Product Management and working together is the best way to ensure that.

So the test manager wants to know what has been unit and integration tested and it's a bit more extra work for the developers but it saves the overall work for the project! By giving the information to the test manager, they can focus their test team effort on what is critical and important. As previously mentioned, if an area of code is not unit tested, the team can focus their effort there compared to an area that is heavily tested - why duplicate the effort? You're all working towards the same ultimate goal of higher quality software released on time.

Improve this answer
1

I would think that it would be beneficial to provide this type of thing. Unit test coverage should be something that is known by development and testing so that they can account for that.

Obviously, you have to test the business critical stuff no matter what. You have to test the commonly used functionality hard regardless of whether it has great unit test coverage. It couldn't hurt to let them know what other places are covered by unit tests. Does the code already check for edge cases in this one little control? This kind of stuff is helpful to know on all sides of the business.

Improve this answer
1
  • I was about to write a similar answer. While unit testing should be in the domain of the software developer, giving the testing team a sense of the code coverage can help the testing team to understand and target specific areas that may require greater attention by the testers. It can also be a way to keep the developers on their game in terms of ensuring they account for as many edge cases as is cost effective to do so. This allows the testing team to not only validate the entirety of the system, but also to account for all of the stuff that might otherwise be seen as costly to test.
    – S.Robins
    Jun 5, 2012 at 23:58
1

It is worth mentioning the approach discussed in the book "How Google Tests Software": Testing and Quality Control is everybody's responsibility, and the standards are rigorous.

The real role of what is traditionally called the "Testing" department, is actually developer productivity; i.e. automation to enable the organization to reach the required level of rigor economically.

Improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.