I have a task to develop a test framework - or a test suite, if it makes more sense - that aims to validate properties over a large set of XML files.

Our codebase is basically like this:

JSON input files -> Python scripts -> XML output files 

Where the Python scripts are responsible for performing complex transformation and calculation rules.

I thought about a couple ways of validating the XML files:

  1. Create a set of rules and assert them only on the output files. This one sounds more like a unit test.
  2. Perform a validation comparing input and output files. This one sounds more like an integration test.

Do these validations make sense? Am I missing a better way of doing it?

  • The absolute simplest way to do this is with a combination of simple unit tests, which test each transformation function against known/human-computed inputs and outputs, and "snapshot" testing, where you essentially do the same thing, but for the whole transformation pipeline. Again, you take a set of inputs paired with human computed (or validated) outputs with that cover your scariest edge cases. It's not difficult. It may be tedious. But, it should not be difficult.
    – svidgen
    Commented Jul 4 at 15:30
  • 1
    I really don’t understand why this question is getting such a negative reception and I’ve been here awhile. So I doubt the OP is learning anything from this. Please consider leaving a comment before downvoting. Commented Jul 4 at 16:35
  • 1
    What is the problem that you are fixing? E.g. "Why do we keep breaking the foo.xml file? It's static content. We don't even need to generate it, as we already have it." or "We keep generating invalid XML files; can't we validate XML generation sooner?" or "We keep changing the generation and some of the files get generated with incorrect values (in valid XML)." Currently, I don't know if you always have the same JSON files, generate them from something, or get them from someone else.
    – mdfst13
    Commented Jul 4 at 21:00
  • Based on your description, do you believe that unit tests don't observe the input that goes into the unit under test?
    – Flater
    Commented Jul 5 at 1:45

3 Answers 3


Do these validations make sense? Am I missing a better way of doing it?

Both ways involve rewriting the transformation code as a test. Sometimes you’re stuck testing that way but I don’t think you are here.

Let me propose a simple regression test strategy that works like this:

compare(transform(test1_json), test1_xml)

Done like this you can have many tests. Keep adding more until you hit all your corner cases.

Whether I’d call that a "unit test" depends mostly on how fast this is and if it involves IO. You don’t actually have to hit the file system. You could do this with strings. But how reasonable that is depends on the size of the json and xml.

The best argument for making this a "unit test" is that transform is a pure function (at least it could be). The test certainly doesn’t have to be an integration test that deals with peripherals. But that depends on how you wrote the transform function and where you're keeping the json and xml. Neither transform nor its test have to know the file system even exists.

I like to separate my slow tests from my fast ones. Mixing them together ruins the fast ones. Call them what you will.

Now if you have an operational need to mess with files all this still works. You just need an outer transform method that calls transform and handles the IO. That outer transform method can be mindlessly simple and so wont require much testing at all, if any, since the business logic is down in the inner transform method and already tested.

This pattern has a name, functional core imperative shell. It does a wonderful job of simplifying testing. Just costs you coming up with names for these methods.


I don't think your comparison of these tests with "unit tests" and "integration tests" matches the popular understanding of these terms.

  • A unit test would be a test which takes a single (partial) transformation/calculation function or module from your Python scripts and validates it in isolation, probably without using any input and output files.

  • Any test which runs a fully combined complex transformation / calculation step, as described in the question, is something I would call an integration test, regardless whether it is using approach #1 or #2.

All of those test types can make sense, and for most scenarios I can think of, tests of all three categories (unit test, integration test of type #1 or #2) will be useful. Note your approach #1 is not suitable for testing the semantical correctness of an XML file, only the syntactical, since just looking at an output file with ignoring from what input it was produced cannot tell you if the content is correct in regards to the input.

Still, which tests make most sense, and on which category one should focus depends ultimately on the specific transformation rules and the specific requirements - which are not mentioned, not even scetched in the question.

And that is the real warning sign to me: my impression is you believe this question could be answerable without knowing more about the semantics of the described processes. The question does not give readers a clue, there is not even a small example, hence I guess you believe this isn't necessary to understand what's going on.

Don't get me wrong, but I think that does not work. Even when you want your testing framework to stay somewhat generically, you still need to analyse your testing requirements first! - maybe not with all the gory details, but at least on a level of abstraction where you have a firm understanding of the problem domain.

Here are some questions you may start with. Ask yourself:

  • are there transformations or calculations which can be tested on their own? Or could be, after some minor refactorings?

  • what are the typical use cases or individual transformations your test code may have to deal with serve?

  • roughly - how many JSON files are required as input for each individual use case, and how many XML files will the case produce? Is it always 1 JSON file which leads to 1 XML file? Or do you expect different numbers?

  • which means are available to verify the semantical correctness of a certain output? Can this be done manually? Are there some tools available, or do you have to create such tools? Will a sample examination be sufficient, or do you need a full verification?

  • are certain transformations reversible, so one might apply the transformation, apply the reverse one and compare the original input with the result?

  • what is the order of magnitude of cases you have to deal with, the typical size of those JSON or XML files and the typical numbers? Are the numbers small enough so test data can be constructed manually, or are they so large you need to think of a programmatic solution?

Find out what kind and how many different examples and data scenarios you need to run by your framework so you feel confident in the code, then you will probably reach the point where you know which of the tests scetched above will serve you most, and if your overall approach makes sense.

  • Not to mention whether or not the Python scripts have already been written and the OP is trying to test them after the fact. Commented Jul 3 at 23:34
  • @GregBurghardt: I guess the OP asks about a living code base. I would expect code which already exists, and code which will be written in the future. The existing parts are a probably a good basis for the missing requirements analysis, but might need some refactoring to make them unit testable. In the end, it does not really matter - even for a generic framework, one needs to have at idea what's going on with these JSON and XML files. Just "they are processed by some Python scripts" isn't enough.
    – Doc Brown
    Commented Jul 4 at 6:43

Checking the validity of XML files has partially been solved. You can descibe the allowed XML in terms of XSD or DTD. Then there are tools which you can use to check the validity of your XMLs against those definitions. Or you can write that yourself in a few lines of code (maybe 40 in C#, not sure about Python).

Potential benefit: you can use that in whatever application that processes the XML files, before processing actually starts. That way you may notice some invalid changes by users who edit XMLs manually.

However, not all rules may be covered by XSD or DTD, so you may still want other ways of checking. So this can only be part of your design. See the other answers for that.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.