Managing manual test protocols [closed]

Question

We have to organize a very long list of manually running tests. Currently we use Word documents, print them out check them off etc. Ugh-ly but works, with problems.

Problems with the current solution

• Requesting and executing subsets, depending on the actual changes. (i.e. tests 153, 155-157 and 24325 need to run only if Module X changed)
• Branching. Different releases (which are still maintained, up to a point) have different test protocols, but share a lot of contents. Manual copy & paste.
• Tracking "Test Template" vs. "Test executed" documents. It's a bit old-style office with lots of forms & binders.
• Tracking in source control together with the source changes. Ideally, the same way features are copied / moved between branches, i.e. as diffable document that is included in a commit.

Content requirements

Documents need images (usually, hardware setup specs and screenshots of "good" curves), "Pass / Fail" checkboxes and minor formatting (suc has emphasis, steps, distinction of "things to do" and "what to expect")

Current idea

One of my devs suggested we use HTML documents instead (Sharepoint Designer is deemed appropriate for editing), "normalize them somehow so they are diffable", and put them under version control together with the sources.

I am personally not wholly sold on this idea for various reasons, but I agree with the other guys that if it works it's better than Word documents.

The Question:

How do you / would manage the documentation for manual tests? Do you know any tools? Ideas?

We are open to other solutions, but only with minimal custom development efforts. Example: installing dedicated software, configuring some templates would be great, but we have no resources for "hack something together with XML and stylesheets and a database, that should be easy".

Some facts&figures

Source control is git, 6 software developers, ~1.5KLoc, one run through all manual tests takes about 4 man-weeks (order of magnitude) and roughly the same in wall time with all modifications and repetitions. (A full run only happens for major releases.)

---

Note: Please don't tell me to test automatically, thank you but we already do. The manual tests involve integration, visualization and external measurement equipment. In addition, the information produced is so multifold that the discoveries of a engineer who knows the whole system discovers a whole lot more with a single look at a chart.

---

Example: A typcial test setup would look like this, every question mark needs to be "checked", it's usually 5..15 tests per setup

Connect 10Ohm resistor to S1. Connect amplifier input to OUT 1. Run test Template 42.

• Measure voltage over resistor. ~1Vrms ?
• In Table 3 Uac = 1.0 V, Iac= 0.1 A, ISNR+D > 45 dB ?
• U(f) Spectrum flat for f > 10 Hz?
• ...

Answer 1

How about moving your test documentation to a format like reStructuredText? You would be able to keep the documentation in git along with the source code and it's easier to diff than Word documents. It allows you to generate the documentation into other formats like HTML and PDF. Regarding your first point about subset testing, in reST it's possible to make documents include other documents so you could probably have "template" documents that pick "test case" documents from different places and generate complete test instruction documents for different modules.

Answer 2

We're using http://www.gurock.com/testrail/ to manage our manual testing. It allows us to create a bunch of different test/templates that can be combined into different test suites and the QAs execute the test suites depending on what type of test they need to do (smoke test, pre-release, post release) etc.

Each test run is tracked separately and you can create reports on different aspects. There's a lot of flexibility of how to structure the individual test, so you can fail either the whole test (e.g. Test 42) or an individual step in the test (e.g. "In Table 3 Uac = 1.0 V, Iac= 0.1 A, ISNR+D > 45 dB ?").

I wouldn't advice any process to rely on shared documents, even if they're source controlled, as you end up spending a lot of time on maintenance. Better go with a tool that's build for the purpose :)