8

This is a follow up to this question. There I was asking how to do unit testing when you have a library of scientific algorithms. I have a similar problem now but with a different project.

I'm working on a 3D graphics engine framework abstraction for DirectX, OpenGl, WebGl, Silverlight, WPF, and basically whatever 3D API is out there, on C#, called Rendering .NET. The scenario is the following, the project is being developed by a small team of colleagues of mine, all working in the same office. We are all computer scientists, not much into the software engineering community and practices. I even had a hard time to convince to switch from Subversion to Git, and to publish the project on Codeplex. The project is getting big (around 80K lines of C#), and it now covers most of DirectX and OpenGl up to Shader Model 5.0, so its not a one man's project like the one described in the linked question. We also want to encourage the open source community to collaborate.

So far we've been testing by writing small applications that involve initializing all devices and resources, setting up a scene, and drawing. The thing is, I don't know how to make it different, I mean, how to design small test cases that test specific features of the framework, like resource allocation, primitive tessellation, shader compilation, etc., without having to recreate the entire engine initialization. For some of these cases I can think of useful mocks, but in general, how can I test the correctness of a rendering algorithm when the output is visual (an image, or an animated scene)? Right now the most clever thing we have come up is to render the same scene to a software renderer, DirectX and OpenGl, and compare them pixel by pixel, but tiny variations of the APIs make the resulting images different enough to fail the tests without being actual bugs.

So, what is the "correct" testing approach here?

Improve this question
3
  • You could implement tolerance levels in your pixel-by-pixel comparison, but that'd be a game of false negatives vs. false positives, and you'd still be partly testing the DirectX / OpenGL implementation, not only your code (so, not really unit testing, per se). In most "normal" scenarios, you'd want to mock the target API, and ensure that it gets called according to your assumptions; if your rendering code is calling the various APIs directly, you could perhaps introduce your own intermediary layer (mockable), albeit at a performance penalty.
    – Daniel B
    Feb 27, 2013 at 8:22
  • I should also add that I believe Doc Brown's answer is essentially correct, you should (ideally) have really independent parts which you can test in isolation. In reality though, if you haven't architected your solution from the ground up for unit-testability, this might be difficult to retrofit.
    – Daniel B
    Feb 27, 2013 at 8:27
  • @DanielB: exactly what I thought when reading that 80K of code already exists. This is legacy code - as Michael Feathers defines it - code without tests. And the canonical book for this kind of situation is always amazon.de/Working-Effectively-Legacy-Robert-Martin/dp/…
    – Doc Brown
    Feb 27, 2013 at 11:57

1 Answer 1

Reset to default
8

The "correct" testing approach would be to decouple your drawing logic from the DirectX or OpenGL calls, so you can mock the latter out (and as a another use case, use the same business logic either for DirectX or OpenGL).

You don't want to test if DirectX or OpenGL is working correctly - that is Microsoft's job. You also want your device initialization code to be written and tested only once (and then reused), so a manual test of that part should be affordable. If you want to test that part automatically, too, use your pixel-by-pixel approach, but with a very simple test drawing. Thus focus on writing automated regression tests for your decoupled drawing logic, that will bring you the most benefit.

So the whole idea is to split your code into really independent components - your DirectX initialization should not be coupled to the drawing logic, and the drawing logic should not depend on DirectX - this makes your program testable.

EDIT: found this former post on unit tests for OpenGL code, there were only a few answers, but perhaps it will bring some additional insights.

Improve this answer
5
  • It seems that this approach would not cover a large part of what this library is all about.
    – Kris Van Bael
    Feb 27, 2013 at 7:20
  • @KrisVanBael: perhaps, perhaps not, you are just guessing. But feel free to make a better suggestion for a testing approach in case you are right.
    – Doc Brown
    Feb 27, 2013 at 7:26
  • Thanks @DocBrown. The framework is decoupled from specific API implementations, through a lot of interfaces, and we are using dependency injection everywhere to solve resource managers, renderers, compilers, and such, so that you can, say, create a mesh with DirectX and visualize it on OpenGL. So you say I should write a kind of "software implementation" that mocks the entire framework and unit test there? Seems nice. I agree that I should not test DirectX neither OpenGL, just my middle layer.
    – Alejandro Piad
    Feb 28, 2013 at 16:51
  • @AlejandroPiad: how to eat an elephant? One bite at a time. Add tests along your development roadmap, testing the parts of your program you are going to change next. Your program was not build in one week, and so your testing framework won't be build in that time.
    – Doc Brown
    Feb 28, 2013 at 17:14
  • 1
    @DocBrown thanks a lot. I think I get it. I`ll start mocking the vital parts first, and try that everything new we add is designed with testing in mind.
    – Alejandro Piad
    Mar 1, 2013 at 15:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.