There is a rather large data flow engine - more than 2000 different flow definitions of "what to do with inbound data". The engine deals with various data formats (flat-file, CSV, JSON, XML, or even binary), performs filtering, transforms data formats, etc.

The flow engine consists of a number of libraries and tools involved in the processing. There are 3rd party ones like Saxon performing XML transformations or Jackson for JSON parsing and a variety of in-house converters, filters, etc.

Naturally, there is a need for deploying new versions of the libraries and tools (new features, security fixes, etc.). This is a risk since correctness of the processing is business-critical.

A simple tests like - unit testing the new feature, integration testing with a fixed set of input data - is not enough. There have been cases of regressions which first appeared after several days after deploy. For example, a very rare combination of numeric values triggered a formatting in a way it triggered a Saxon optimization bug which caused the formatting being omitted.

The currently employed method of testing is to compare new and current versions "online" for several weeks. It's like adding a "copy & divert" stage resulting in processing the inbound data using both the current and the new version of a tool/utility and then comparing results. This is a very cumbersome, time consuming and potentially risky.

I've been thinking about a more effective way of doing this. Any ideas?

  • Why exactly is the process cumbersome and time consuming? Maybe there is some potential for automation?
    – Doc Brown
    Commented Sep 18, 2020 at 13:22
  • Of course it's being ran automatically. However, there are many things you need to check. An example: You wanna test a new JSON lib, but there is a number of flow steps enriching JSON data with a timestamp. You can't simply compare the data as a whole since your prod and "diverted test" will both generate a different timestamp. You gotta ignore the timestamp attribute or mock that one out.
    – Yuri
    Commented Sep 18, 2020 at 13:28
  • Ignoring the timestamp or mocking it out is probably the way to go. Surely you have to invest some time into those details, but the idea should be exactly that: elimininate anything which hinders the reproducibility of regression tests, so you can automate as much as possible. Your general strategy looks fine, now you have to optimize, optimize, optimize.
    – Doc Brown
    Commented Sep 19, 2020 at 8:51

1 Answer 1


There is only so much you can do, and it sounds like you are already doing most of it. In particular, your "copy & divert" method already goes above and beyond what most people do. I've only heard of that being used in extreme situations like spacecraft. One thing you could do is instead of doing this test online, record the inputs and outputs and run it offline all at once.

You may already be doing this, but you should definitely try to create or move particular tests to earlier stages when you detect regressions. If you find a problem in production, write an integration test so you can detect similar problems in the future. If an integration test finds an issue, try to write a unit test that detects it.

The other thing I didn't see you mention is fuzz testing or quickcheck-style testing. These have the computer generate test cases for you that you may not have considered.

  • The problem with gathering and storing the full data sets (like a month of traffic) for offline testing seems to be the compliance. Storing this large amount of non-anonymized data for the internal testing purpose sounds a bit scary.We create unit tests for each regression we discover in our SW. However, the regressions from 3rd parties (JRE, libs, etc.) are outside of this scope. Will check the Fuzzing and Quickcheck!
    – Yuri
    Commented Sep 18, 2020 at 13:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.