What are the best practices for managing Test results

Question

We are using GitHub for managing source code and waffle board for managing workflow/issues.

Right now when we test the system using custom written test cases, it generates a CSV file. We want to be able to keep a record of these test results, so we can go back and run the same test again with the same Inputs and verify the results, or just share the results with stakeholders.

Whats the best strategy of managing these test results?

Should we publish the Results.csv's in the same Github repo as the project? (that would become cumbersome and we want to avoid that)

We tried publishing results in Waffleboard, but Github issues don't support File Uploads for us to attach the results (only Image files can be uploaded)

The only option we see is publishing the results on an internal website. Is that the best way of going about this?

Edit Clarification:

The system is replacing a legacy system. The test cases change daily.

The test script grabs the data from legacy system and from the new system, and does a comparison to see if they match.

Answer 1

The test results should be managed in the same git repository. You don't need to save results of every run, you only need to save the results for a single run. All other runs should compare itself to this "golden" version of the data. If a test generates different results, it will have failed.

Put another way, this .csv file isn't a test output, it's a test input. Generate it once, then your tests should use it to validate whether the current system is performing as it should. Since it's an input, it needs to be version-controlled just like any other test assets.

When you run your tests, you can create a daily report that shows only the failures. There would be no need to archive this unless you need to do analysis as to the frequency certain tests fail or pass.

Answer 2

From your question:

We want to be able to keep a record of these test results, so we can go back and run the same test again with the same Inputs and verify the results – @user3711455

From your comment:

The .CSV changes daily. The test script's input is the output of a legacy system. It takes that, gets equivalent data from New system, and then compares to see if they both match. Resulting with a CSV that has: Legacy System Results, New System Results Total Differences: – @user3711455

These are not the same test. Running the same test, with the same inputs, against the same code should be a redundant exercise. If that doesn't always produce the same result you've allowed magic into your system.

This is useful only in verifying nothing magical is happening. More typically you rerun these tests making sure only one thing has changed. Usually refactored code. That way when the test breaks you know what to blame, the one thing that changed.

Running two systems side by side duplicating work for comparison is not a test. It's a voting system. When they disagree you have to decide who to believe. The legacy system shouldn't be trusted to be perfect no mater how many years it has under it's belt. In fact the older it is the more likely some mistakes it makes are so well known that everyone just ignores them since they are expected and may have never told you new guys about them since everyone knows that. You may get some value out of this proving the new system is ready to transition to operational but these are not tests since they require the old system to exist.

However, feed input to the old system and record the output and you have a baseline output. But just for that input, and just for that version of the legacy system. Which hopefully isn't buggy itself. From that you can build a test the new system. But if your inputs don't exercise every use case and every bit of code then you're just hoping to get lucky.

If, as I suspect, your custom written test cases that cause the test cases to change daily is just having both systems work from the same feed of operational input I don't have a lot of confidence that you have good code coverage.

If, however, you're input is hand tuned to exercise different parts of your code then those inputs and the code that automates testing them and comparing their outputs should be organized in a structurally similar manner to the code that they test so that it's very easy to navigate from one to the other. Do that and keep it all version controlled and you'll have a nice development system.

Answer 3

Although it answers a different question, I think my answer to a question about storing the results of static analysis may be useful here as well.

You need to version control your test cases and test input. Given a software version, you should be able to exactly reproduce the tests that you ran and their associated input data at that point in time. However, you may also want to be able to run newer tests on older code, as well. Keeping your tests (test code, test input data, test procedures) in your version control system and tagging them is essential.

Once you are capable of rerunning the exact same tests with the same data over a snapshot of the software at any point in time, I don't think it's required to keep all of the test results.

If you feel the need to make some data available, you can package up the test results as part of a formal release. If you include the test results, you can look at how you want to package it - including all of the test outputs and results or writing a summary report that describes the test results.

Answer 4

We use three types of tests:

• Selenium
• NUnit
• Soap UI

All of these produce result files (XML). For the Selenium tests, these are high level flows so we send an email to a group who can view the results since there are less than 10. These are pretty much technical check out style of tests (smoke test). The XML is converted to HTML and pushed into the email.

For the nUnit and SOAP UI tests, we have 100s if not 1000s. Both of these test produce XML files that are nUnit test result format. From there, we feed these results to a simple web page (Report Unit) which converts the XML and then puts the results to a web page.

We don't care about past results, but you could push each "run" into an archive folder for review, but results from even a few days ago are stale from our point of view and may no longer be valid if there are many code changes. These results are sent to a monitor (TV) so the entire team can see the health of the tests. Tests that turn red are investigated and resolved.

The mechanism(s) to produce the results are under source control, but the actual results themselves are not in source control.

Answer 5

Best practice allows a developer to execute the test suite from a fresh working copy without external dependencies.

Your scenario already seems to diverge from the typical testing scenario. But, what that probably means for you is storing the "target" results in the repository. Your testing tool shouldn't store its immediate results anywhere; it should compare its immediate results against the "target" results for validation and report individual failures.

From there, any single failure should prompt the developer to either fix the bug or validate the change in requirements and adjust the "target" data accordingly before committing.