Background: I am thinking of trying to introduce the concept of unit tests to my co-workers by creating some for a module I have been working on; the requirements of it recently changed and require some more abstractions/interactions so it seems like a good way to develop a suite of tests that will "prove" it works without having to manually poke around the application.

The issue, however, is that the module relies on unmockable external factors namely PDFs and XSL. Basically I read XML from the database and apply an XSL transformation to it, then convert it to a PDF using a library called ABCPDF. This PDF is then merged with another PDF based on a static template. I know that I can test the XML and make sure the values are correct, but many of the potential bugs and issues are related to the actual display of the finished document - e.g. minutiae like how long text strings are wrapped, where certain HTML areas are located in relation to the document, etc. Is it even possible to test these things (I realize these are probably integration tests or.. the third kind of test whose name I forget [not Acceptance tests, the other kind], and not unit tests) since I can't, to my knowledge, mock out a PDF easily short of creating it then reading it back or creating an HTML string (i.e. transformed XML) and parsing it by hand to check for the presence of certain table cells in relation to other table cells.

In a situation like this should I just focus on unit tests to make sure the information is correct and that I can create the PDF, or merge them, or whatever and resort to manual testing for the actual display issues?

  • 6
    "unmockable external factors" is a hint that you're not doing unit tests in the first place. That means integration testing. Which do you want to do? Unit testing of your code or integration testing of this composite thing? Please choose one or the other, because it's hard to talk about both at the same time. – S.Lott May 20 '11 at 15:55
  • 2
    I'm not buying "unmockable". I will accept "that I don't know how to mock", which just means that your real question is "how do I mock it?". – Rein Henrichs May 20 '11 at 18:32
  • Probably :) I'm familiar with mocking the XML used, but not with how to mock an actual PDF or an HTML document where the formatting matters. – Wayne Molina May 20 '11 at 18:38
  • 1
    I think you mean "functional" (application end to end) or "system" (multiple applications end to end) tests – Gary Rowe May 20 '11 at 19:01
  • @Gary - Yes, functional was the word. I remember them now from learning Rails: Unit tests models, Functional tests controllers, Integration tests everything. – Wayne Molina May 20 '11 at 19:13

Test the feature not the unit

Using known xml inputs, output a PDF and manually (and meticulously) verify that it is correct. Then save it as a reference.

Future tests using the same xml inputs can do a binary-file comparison to the reference.

If a file-level comparison is unsatisfactory, display the PDF at the end of the test and take screenshots, then have the automated test compare to the reference screenshots.

  • +1 because you're only interested in the final result at this level. If you change the implementation of how you arrive at getting the PDF you shouldn't have to change your functional test. – Gary Rowe May 20 '11 at 19:01
  • 2
    +1 for good advice, this is what we do in our current project. We built a custom toolset for doing the PDF comparison, which allows us to omit changing parts in the documents such as timestamps. Caveat: switching to a different (version of the) PDF renderer may cause subtle changes in the layout, causing direct binary comparison to signal heaps of false positives. – Péter Török May 20 '11 at 20:39

Normally in a case like this you abstract everything you cannot test behind an implementation you can use with an interface. I'll just do something silly like PDF builder because that seems reasonable.

public class PdfBuilder : IPdfBuilder
  public byte[] BuildPdf(...)
    // actual untestable code here

public interface IPdfBuilder
  byte[] BuildPdf(...);

You can then mock the IPdfBuilder in your tests to do whatever you want. This often means you need to start using an IoC container (https://stackoverflow.com/questions/871405/why-do-i-need-an-ioc-container-as-opposed-to-straightforward-di-code and https://stackoverflow.com/questions/21288/which-net-dependency-injection-frameworks-are-worth-looking-into as place to start) if you aren't using one now.

And tests that are not unit tests are often called integration tests. Complicated integration tests are often not totally worth it, so you just abstract out that part and reduce the amount of business logic in that abstraction so you can test it in a unit test.

Let me know if this isn't totally clear.

  • +1 for hiding untestable code. Then you can do manual tests until you work out what needs to cross that interface to get the right result, and unit test for that being generated properly to get your regression unit tests. – Ethel Evans May 20 '11 at 19:32

I built something very similar a while back, and just used basic visual tests. Testing does not have to be automated, so there is nothing wrong with just looking for an expected result (obviously, in a variety of situations that are predetermined). Often, a picture is worth a thousand tests where visuals are concerned. I use automated unit testing extensively, but I think some people can get a bit carried away when getting into GUI testing, or anything visual IMHO. With certain product, I do recognize that this "good enough" approach will not be sufficient so YMMV.

I would be a bit worried, however, about the unmockable externalities. This can be a sign of tight coupling, which is good to avoid as a general rule of thumb, but I won't speculate too much on your code in that regard without more details.

  • It's very tightly coupled but that's an area I can't fix since there's no buy-in to make it loosely coupled and no resources are devoted to refactoring (but that's a whole different set of issues). – Wayne Molina May 20 '11 at 18:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.