Report test failures "all at once", or report them one by one

Question

I am a sole developer working for an organisation that has no testing strategy.

I am trying to integrate a crime system developed by a vendor with an external application I developed. The external application allows an end user to send an instruction to the crime system to delete the crime.

The crime system was developed by an external organisation and they have written stored procedures to actually execute the deletion, which I can call from my application.

The crime system is complex i.e. it has a custody element, a crime element, an intelligence element etc.

Whenever there is an upgrade to the crime system the stored procedures have to be retested and there are always many issues. My approach is as follows:

1. Test custody
2. Report custody issue 1
3. New release by vendor
4. Report custody issue n
5. New release by vendor. All tests pass.
6. Test crime
7. Report crime issue n
8. New release by vendor. All tests pass.
9. Report intelligence issue n
10. New release by vendor. All tests pass.
11. Testing complete. Goes live.

The vendor has asked me to do all testing at once in future. In my experience this does not work very well with this vendor because new issues are often introduced, which affect testing to do in future i.e. a custody change affects the crime element/intelligence element.

Should I be testing everything and then submitting all of my findings?

Answer 1

If company A develops a individual software component for company B, part of the contract is typically that B defines or develops a list of acceptance tests, and B will only pay A if the test do not show any issues. That is perfectly fine.

However, if some of the tests fail, B must report them to A, and A gets a chance to add corrections. And since every correction has a certain risk of breaking things which already worked before, B will have to repeat all of the tests again. So I understand your question as how to make this process most effective for both of you.

The first thing you should try is to automate as many of the acceptance tests as possible. Since you have to repeat the tests over and over again, it should be obvious why. Running the test suite with all tests at once should produce a big log file you can send to the vendor. Of course, the log may show some consecutive faults, but that should be the vendors concern, not yours. The more feedback you can give them in one cycle, the more bugs they can fix in the next release. Hence, what the vendor told you sounds sensible to me, if that is the situation.

However, things become more complicated if you cannot automate all of the tests, and the testing procedure needs a lot of manual effort. Maybe the whole process will take some days, and not just a few minutes. You wrote you have to integrate the component with an "external application" of yours, so I can imagine even if you have automated tests, you may have to check manually that a test failure is not caused by a bug in your application.

Now, if you already know for sure that if part X of the component fails, part Y will also fail, because Y depends heavily on X, then it will not make much sense to start any laborious tests on Y before X runs smooth. Thus it will obviously be more effective for you to give feedback about part X to the vendor and request a bugfix, before you start with the tests of part Y.

In such a situation, the best advice I can give you is to find an agreement with the vendor where you explicitly name the parts X1, X2, X3, ... you are going to test first, and tell the vendor that only if those tests are passed, you start testing parts Y1, Y2, Y3, ... If the vendor behaves reasonable and knows about the dependencies in his component, I guess you can find a consensus about this.

Answer 2

Wow! They're contracting you to do their testing for them.

The best solution is to implement an SLA that gets you out of this relationship. Your contract should penalize them financially for bugs you discover that impede integration of their product.

I really don't have anything to add on what is "best practice" when working within some relationship where they're managing you and you also pay them. Send them bug reports by carrier pigeons, maybe they'll put you on performance review.

Obviously finding a new vendor is an option I could consider. Or in-house the solution, considering you already are in-housing the QA team anyway.