TDD for batch processing: How to do it?

Question

I like "red/green/refactor" for RoR, etc. just fine.

My day job involves batch processing very large files from third-parties in python and other custom tools.

Churn on the attributes of these files is high, so there are a lot of fixes/enhancements applied pretty frequently.

Regression testing via a known body of test data with expected results does not exist. Closest thing is running against the last batch with new test cases hand coded in, make sure it does not blow up, then apply spot-checking and statistical tests to see if data still looks OK.

Q>> How to bring TDD principles into this kind of environment?

Answer 1

Just an FYI: Unit testing is not equivalent to TDD. TDD is a process of which unit testing is an element.

With that said, if you were looking to implement unit testing then there's a number of things you could do:

All new code/enhancements are tested

This way you don't have to go through and unit test everything that already exists, so the initial hump of implementing unit testing is much smaller.

Test individual pieces of data

Testing something that can contain large amounts of data can lead to many edge cases and gaps in the test coverage. Instead, consider the 0, 1, many option. Test a 'batch' with 0 elements, 1 element and many elements. In the case of 1 element, test the various permutations that the data for that element can be in.

From there, test the edge cases (upper bounds to the size of individual elements, and quantity of elements in the batch). If you run the tests regularly, and you have long running tests (large batches?), most test runners allow categorization so that you can run those test cases separately (nightly?).

That should give you a strong base.

Using actual data

Feeding in 'actual' previously used data like you're doing now isn't a bad idea. Just complement it with well formed test data so that you immediately know specific points of failure. On a failure to handle actual data, you can inspect the results of the batch process, produce a unit test to replicate the error, and then you're back into red/green/refactor with useful regression cases.

Answer 2

Its the same as any other environment.

Separate out the logic into its smallest level of granularity. This will give you a set of rules for the process, each rule will cover one item of logic that is required for your process.

Then write a test for each rule. These tests will fail. Write the code to fix the test.

The regression testing with known test data that you are talking about is not unit testing. That would be integration testing, this is different from TDD. With TDD you may have a single test to test that you could load a file, but generally no other test would actually go near a data file with test data. Instead you would simulate the data required to exercise a particular rule using a mocking object.

Answer 3

Start with a good software strategy, then apply TDD.

(Disclaimer: I may have misunderstood "churn", or TDD, or both.)

Here is my suggested strategy for batch processing "dirty data": Specify-Triage-Execute.

• Draft a specification of the data in a stringent and narrow way, yet will cover the majority (say, 80% or more) of the incoming data. Call this Specification 1.
• Develop a Triage module (TDD if you wish) that decides if a record meets Specification 1.
  • Make sure the module runs very fast.
  • The module should return true/false: it either meets all of the rules, or it doesn't.
• Develop an Execute module (TDD if you wish) that parses a record that is known to meet Specification 1, performing whatever tasks needed by your customers.
• Apply Triage 1 on all incoming data.
  • The result is one boolean value for each record. This basically separates the incoming data into: Specification 1, or Unknown.
  • Apply Execute 1 on the Specification 1 data, whenever needed by the customer.
• Relax the rules of Specification 1 in order to admit 80% of the remaining data. Call this Specification 2.
• Develop Triage 2 and Execute 2. Apply it to any data that did not meet Specification 1.
• Repeat for as many levels as needed, until the remaining data is small enough that it can be manually processed every day.

The efficiency tidbit:

Save all of the Triage results (historical or current) associated with a record. If no Triage modules are modified since the last run, then it does not have to be re-run on old data.

The "you have to know what you want to build before doing TDD" tidbit:

The Specify-Triage-Execute is one way to keep the requirements manageable on each level and allows for future growth.

(If any knows the standard correct terms for those three steps, please let me know, I'll edit my answers.)