-1

We have a logging protocol that defines how output of logging libraries should look like, e.g.:

  • each line should be a JSON
  • each line should not exceed 1MB

There are multiple implementation of this logging protocol, i.e, logging libraries written in different languages (Java, Python, etc.). I would like to create one set of tests that I can run against any logging library to verify it complies to the logging protocol. I don't want authors of a new library to port existing test. Instead, I want them to reuse existing tests.

The are already known solutions to that like Reactive Streams Test Compatibility Kit, but they work for implementations in the same language (or at least for JVM-languages). In my case implemtations can be of different languages.

How can I make tests language independent?

Initial idea

  1. My first idea is based on assumption that logging libraries always take one or two arguments:

    logging.error("Serious problem occurred", exception)
    

    and log output to a log file

    { level: "error", message: "Serious problem occurred", stacktrace: "....."}
    
  2. The test harness would pass some string to the testing library and verify the output.

  3. The test harness would communicate with a logging library through a pipeline, so the only thing that the creator of the library need to provide is a wrapper for handling communication with a test harness (through pipeline).

Is there a better approach?

5
  • 1
    What do you mean by "better?" Nov 13, 2017 at 17:39
  • How does the infrastructure look like which runs the test? And about how many languages/libs are you really talking?
    – Doc Brown
    Nov 13, 2017 at 20:29
  • @RobertHarvey I should've propably asked what are the limitations of my solutions. Also, I don't want to re-invent wheel: I am sure that the idea of standard/protocol and verifier for implementations of this protocol is quite a generic problem, e.g., XML verifiers, TCK for Reactive Streams. So I am looking for similar solutions to learn from.
    – dzieciou
    Nov 14, 2017 at 12:36
  • @DocBrown So far, only three libraries, in: Java, Python, and node.js. Tests are run on Jenkins, but I haven't thought about test runner yet. I thought that my test harness would 1) generate input, 2) call underlying library through some wrapper, 3) receive output.
    – dzieciou
    Nov 14, 2017 at 12:39
  • You might be interested in the architecture of Android's (AOSP's) Compatibility and Vendor Test Suites.
    – RubberDuck
    Nov 18, 2017 at 23:47

1 Answer 1

5

Language independet tests are entirely possible, but the problem is that you need to define a language-agnostic format for these test cases, and write parsers/test drivers for this format in each language. It could happen that such a test driver would end up being more complicated than the protocol you are testing.

Such language-independent tests may use Cucumber-style specifications, an ad-hoc format, or a markup language like XML or YAML.

In your case, a simple YAML test case might look like:

---
name: Simple error
error: Serious problem occurred
output:
  json:
    level: error
    message: Serious problem occurred
...

Now unfortunately this test case is extremely weak because the input and expected output are very similar. But it's a start.

You will also have difficulty abstracting over language-specific properties, such as:

  • the format of stack traces (which some languages like C or C++ don't have in a standardized fashion)
  • injecting and asserting language-specific objects such as exception objects (which some languages like C and to some degree C++ and Perl don't have)

Instead of writing language-agnostic test cases, it may be far easier to write a short but clear specification of your protocol. Then, all implementations can be (manually) reviewed for conformance. An implementation may of course still have individual test cases.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.