We develop small to large visualization projects for different tasks and industries and sometimes while rewriting them a couple of times in the process we hit walls because we discover that we need to add a lot of code to support new requirements. Now we have established a design process that seems to work well (at least we reduced the development time for each new project quite a bit), but we're still left scratching our heads around this question: what exactly should we test when testing visualizations?

  • If everything that we want to explore is on the screen (bounded visualizations)?
  • If the data is ok - if data is valid (that's one of the nice things about visualizations you can spot errors in your datasets)?
  • Usability?
  • User interaction?
  • Code quality?

I can tell you for sure that a simple check of the code quality is certainly not enough! Is there a classic paper / book about how to test visualizations? Also do you happen to know about classic design patterns for visualizations (except the obvious ones like Pub-Sub)?

2 Answers 2


What are you "testing" for? Are you testing for accuracy of the visualization, in other words, whether the visualization matches the system it describes? In this case, consider a View Model: an object structure identical to the visualization. Write code that compares it to either (1) the view model to the system it describes or (2) a known correct visualization for a certain system and repeat for as many different visualizations as required.

If you want to test the visualizations themselves, specifically how they appear on the screen, this is more challenging. You can "screen scrape" (compare the bitmap representation) but this could be foiled by changes in rendering engines between OS or library upgrades among a number of things. Depending on the graphics library and operating system used, a GUI testing library may be available to help. See http://en.wikipedia.org/wiki/List_of_GUI_testing_tools for a list.

Are you testing whether the visualization is at the appropriate level of abstraction or whether it makes sense to customers? This is not really "testing". This is more UI or visualization design and is a much bigger topic.

  • You correctly identified some of the challenges. I really love your answer. Can you provide more details about "GUI testing library"? Did you used something like this?
    – paxRoman
    Sep 14, 2012 at 17:05
  • I guess you are talking about something like this: froglogic.com/squish/gui-testing/editions/web.php . Thanks for the idea!
    – paxRoman
    Sep 14, 2012 at 17:07
  • 1
    @paxRoman There are many, many such tools and I have added some details to the answer. I have used Selenium and WatiN for web work before but nothing for native GUIs for several years.
    – akton
    Sep 15, 2012 at 1:15
  • thanks! we would like to test visualizations in the browser - therefore we really need to check for browser bugs and we need a normal Web testing environment...
    – paxRoman
    Sep 15, 2012 at 10:07
  • 1
    @paxRoman Then consider Selenium (seleniumhq.org) or WatiN (watin.org). Selenium (selenium-grid.seleniumhq.org) can parallelize testing if needed, too.
    – akton
    Sep 15, 2012 at 11:14

You could read through the excellent treatise The Grammar of Graphics, which elaborates on different properties of visualizations and how they convey or obstruct the flow of information to the observer. Although the book does not list a test plan for visualizations, it certainly brings to light to connection between code, model, media and perception. For me it was a most inspiring read.

In my opinion, visualizations are neither inherently right nor wrong, but there are certainly best practices. Eg., having an understanding of the perception of color will help you find good ordinal color scales.

  • Yes I read most of the important books and articles from Edward Tufte, Stephen Few, Leland Wilkinson, Jeffrey Heer, Mike Bostock, John Resig, and many others...including books about art and psychology by the likes of Rudolf Arnheim, Ernst Gombrich, Eric Kandel...None of them however insists too much on testing visualizations...:( exactly as you said...it's like they're not good or wrong...but some of them certainly have bugs and we need to fix them and just classic software engineering doesn't apply because there might some other subtle bugs...
    – paxRoman
    Sep 14, 2012 at 8:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.