2

Pandoc is a command-line tool and Haskell library for converting between many different markup and document formats. One of the ways Pandoc's behavior can be customized is via filters -- Pandoc serializes its' representation of the document into JSON and passes the JSON into the standard input of the filter; the filter can return modified JSON to Pandoc via its' standard output, which Pandoc will then use to create the output document. Filters can be written in any language which can process standard input/output.

I've written a library for writing such filters in .NET. In order to ensure the library produces the proper JSON, I have the following test:

  • For a given document, call Pandoc on the document to produce the JSON equivalent; pass the JSON into a dummy filter which doesn't do anything; and ensure the JSON output is semantically equivalent (source). This ensures the library itself doesn't introduce any unwanted changes to the JSON.

I have four dummy filters -- different variants on the base filter classes in the library -- which I am using for the tests.

I run this test against all the (relevant) documents in the Pandoc test documents folder. All the tests pass (save for those documents which Pandoc for one reason or another cannot parse).

But my problem is that running these tests on my machine takes almost 40 minutes, which feels far too long.

I don't want to store the generated JSON for each document (instead of having Pandoc produce it each time), as other versions of Pandoc might produce a different JSON.

Is this length of time a valid concern? What might I do to improve the test pipeline?

Improve this question
1
  • Whether this is something to be concerned about really depends upon the impact. I've worked with test suites which take much longer than this in-full, however the full complement would only ever run within a dedicated environment and on a regular schedule to minimise any impact on the development team. Naturally this causes some delay into QA or Acceptance testing, but it's usually negligible enough for a business owner not to care very much (A short delay to a release process is usually more than compensated by the quality gains of having a thorough suite of automated regression tests).
    – Ben Cottrell
    Jan 3, 2021 at 14:06

1 Answer 1

Reset to default
3

Often times for higherlevel specs I would mock out the foreign thing to test different possible cases and then have as few as possible full integration tests just to verify that everything works together.

I am not familiar with Pandoc but I assume you can peg it to a version if you are using it on your project. I would run a suite of integration tests to ensure that what was written will work correctly from end to end. Then I would have lower level tests that mocked pandoc and just tested that I am passing the correct information to it via a mock or something. That should run pretty fast.

Everytime you bump the version of pandoc, you'll want to run your integration tests to see if anything broke. Normally if they stick to semantic versioning minor number changes won't break, though you never know.

Improve this answer
6
  • In theory, as long as the JSON input produced by Pandoc can be serialized into the types defined in the library, and those in turn can be serialized back to JSON which is parseable by Pandoc, anything goes.I don't really see that I have anything to test against a mock of Pandoc.
    – Zev Spitz
    Jan 5, 2021 at 22:10
  • You had mentioned it taking a silly amount of time. 40 minutes is in fact a long time depending on the frequency of your tests. I guess the real question is "where is the slow-down coming from?" My standard case is a webserver, the db is typically the slow point, so for the tests I run most frequently, I will avoid using the db, and then do lighter tests on integration. This way I can test my business logic thoroughly and fast. Maybe Pandoc is not your bottleneck.
    – unflores
    Jan 12, 2021 at 10:41
  • Pandoc can output its' internal data structures in two formats of interest for testing: JSON, and a custom serialization format. For each test document, my tests involve ensuring that for both formats, the output is the same whether Pandoc produced it with or without running it through the filter program. IOW, Pandoc(document) -> JSON produces the same JSON as Pandoc(document) -> dummy filter program -> JSON; and Pandoc(document) -> AST produces the same custom-serialization as Pandoc(document) -> dummy filter program -> AST. The bottleneck here is in starting up and running Pandoc.
    – Zev Spitz
    Jan 12, 2021 at 12:47
  • There are four variants of the filter program that I need to test; but I have been able to test two variants within one program instance, which halved the testing time down to 20 minutes. I can also cache the produced JSON for each document, and feed that JSON into the filter program; instead of calling Pandoc multiple times to produce the same JSON for a given document.
    – Zev Spitz
    Jan 12, 2021 at 12:50
  • 1
    "dummy filter program" means a program that takes in Pandoc-produced JSON, deserializes it into objects, serializes those objects back into JSON and returns the JSON to Pandoc for output. The purpose is to test the deserialization/serialization, and to ensure no other objects are introduced along the way, by verifying the input and output JSON are equivalent. I suppose this qualifies as an integration test.
    – Zev Spitz
    Jan 23, 2021 at 19:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.