18

I'm writing a game and the accompanying graphics engine on top of OpenGL in C++. Im also a fan of good coding processes and automated testing. Graphics code + testing seems pretty immiscible, since the output is often visual only, or very heavily visual-oriented.

For example, imagine analyzing the raw image-stream that is rendered to the screen byte-by-byte - you need test-data to compare with, which is hard to create/obtain, and often the rendered images aren't identical on a byte level when running at different times - small changes in algorithms will wreck this approach completely.

I'm thinking of creating a visual unit-test suite, in which I can basically render different test-scenes, showing stuff like shadow mapping, animation, etc etc etc. As part of CI, these scenes would then be rendered to a video file (or possibly leave it as a executable) with different metrics. This would still require manual inspection of the video file, but atleast it would be somewhat automated and standardised.

What do you think? I'm hoping there are better ways?

Improve this question
3
  • 2
    This is a good question, I'm wondering if you'd get better answers at gamedev.stackexchange.com since there might be more people there who have dealt with this before.
    – FrustratedWithFormsDesigner
    May 29, 2012 at 15:15
  • 3
    Since you use OpenGL, consider studying the way how Khronos group performs conformance testing for this API. Colleagues in my ex-project who dealt with it told me that Khronos graphics testing is as close to perfect as it gets
    – gnat
    May 29, 2012 at 15:19
  • @FrustratedWithFormsDesigner Thanks, I completely forgot about gamedev.stackexchange... I searched programmers pretty thoroughly, but not that one. It seems most of the answers on unit testing over there focuses on the usually tested stuff though, so maybe this isn't a duplicate. Will check some more though, thanks :)
    – Max
    May 29, 2012 at 15:21

3 Answers 3

Reset to default
8

The opencv image processing library does it by saving the image and comparing it to a reference image - it has a bunch of c++ test functions and macros to handle approximate image matching etc.

Improve this answer
2
  • I'll try to check it out more in depth myself, but that page doesn't say anything about the approximate image matching. Do you know if it would work for regression testing in the case of comparing frames? Then I could visually inspect the first time a scene is rendered, note it as a reference so on. Does OpenCV have ready functions for this I could use? :p
    – Max
    May 29, 2012 at 15:46
  • @max generally there is a 'gold' version of the test function that generates the reference frame and is run once when creating the test. In the distribution there is are a few "test image compare" macros. I don't think there is anything specific to read back openGL buffers to an image but that's easy enough
    – Martin Beckett
    May 29, 2012 at 15:51
4

Your test framework can render its test image into a buffer, retrieve the rendered image, and compare it to a "golden" reference image which has been previously generated for that purpose.

This will not work as well for cases in which the results of your test are not expected to remain exactly the same. However, you can compute the squared difference of the test and reference images, and compare it against a threshold.

You may also want to provide for logging and checking performance data, since a major regression in performance is another possible failure mode.

Improve this answer
4

Even if you can't compare the output images, at least you'll be able to test that your renders complete appropriately (no crashes, long waits, etc, etc). Finding a way to check the images is better, of course, but even without it you'll have gained something from the tests.

Improve this answer
1
  • 2
    Exactly: I was about to answer something similar ("Even if you have a unit test that just blindly generates the video file and does some trivial tests (e.g. has the right length, not completely black, not completely white), it might be enough to catch many regression errors. Especially those which simply cause your functions to throw an exception.") but there is no need for a second similar answer, so I leave it just as a comment here.
    – user281377
    May 29, 2012 at 19:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.