Pandoc is a command-line tool and Haskell library for converting between many different markup and document formats. One of the ways Pandoc's behavior can be customized is via filters -- Pandoc serializes its' representation of the document into JSON and passes the JSON into the standard input of the filter; the filter can return modified JSON to Pandoc via its' standard output, which Pandoc will then use to create the output document. Filters can be written in any language which can process standard input/output.
I've written a library for writing such filters in .NET. In order to ensure the library produces the proper JSON, I have the following test:
- For a given document, call Pandoc on the document to produce the JSON equivalent; pass the JSON into a dummy filter which doesn't do anything; and ensure the JSON output is semantically equivalent (source). This ensures the library itself doesn't introduce any unwanted changes to the JSON.
I have four dummy filters -- different variants on the base filter classes in the library -- which I am using for the tests.
I run this test against all the (relevant) documents in the Pandoc test documents folder. All the tests pass (save for those documents which Pandoc for one reason or another cannot parse).
But my problem is that running these tests on my machine takes almost 40 minutes, which feels far too long.
I don't want to store the generated JSON for each document (instead of having Pandoc produce it each time), as other versions of Pandoc might produce a different JSON.
Is this length of time a valid concern? What might I do to improve the test pipeline?