I am working on a testing tool that provides image comparisons. For test data insertions, I am using excel and csv files that supply the tests with text that is used by the tool to mock actual user interactions - in filling up of forms/screens in the tests. After the insertion, there is a need for verification to ascertain whether the test has functioned properly or not.

What mechanism should I apply to ensure proper validation for the tests ? As the screens will differ widely, I cannot use a hard coded error/success validation string/image.

One approach that I am considering is generation of images based on inputs(OCR) for different tests, which will be used to validate the instance of test where the specified input was executed.

Can anyone comment/share any other technique for doing this ?

  • 2
    My personal experience with image-based testing is that it is not suitable for the scenario you described. – rwong Jul 27 '12 at 13:55

I suggest developing (or even better finding) an algorithm to compute how different the images are. Then have the assertion specify the maximum tolerance for how different those images are. It's not a fool proof approach, but I suspect it'll work pretty well. Similar to the tolerance that can be set by many tools when comparing two floating point values.

Another thing to try would be to allow your tool to define a rectangle within the images that the test cares about. That might help tests specify the exact area that's being evaluated by the test.

| improve this answer | |

What mechanism should I apply to ensure proper validation for the tests ?

That depends entirely on what toolkit and language you're using, and on how portable you need the test to be. Were I going to build such an approach, I'd create at least two parts, one of which runs the test and one of which pre-builds an image of what I expect the test to be, and then I'd do a straight equality test between my generated image and an image of the form post-test. No OCR, no tolerance for variation -- the test suite would build an image itself, and compare it to the image from the actual app.

(I'd also likely kick up a storm if I was asked to do image-based testing, and explore using or building a UI-aware test framework instead.)

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.