TDD and Writing Tests for File Processing

Question

These questions are very similar: TDD for batch processing: How to do it? and TDD: how to test file outputs? but I'm still left wondering some things

The app processes files/pdfs. It takes them from one FTP server, processes/reads file content, places them in a destination FTP server, depending on content read.

How can I approach file processing in a TDD way? My formal education on TDD was not the best, so even "obvious" suggestions will help me.

My ideas:

The "success" of my app depends on if it reads/processes a source file properly, and also if it places it in the proper destination. So it makes sense to me to write tests for seeing if a file

1. is "read" correctly
2. ends up in the right place

I could test on specific dummy files, but does this mean I should always ship these specific files with the code, for the tests to pass? The file's name will be different after being processed, so I'm not sure how to test if it ends up in the right place -- should I eyeball the files and then hardcode expected results into the tests?

Answer 1

I think this is a great question. Unit testing, micro-testing, TDD, and BDD are difficult, sometimes confusing, and often not taken seriously. So kudos for trying and asking for help!

I am by no means an expert, but I did just take a formal class and am also part of a small team that is leading an initiative to roll out unit testing and TDD within our organization.

Based on my experience and what I learned in my class, here is what I think you need to know in order to start:

1. Consider that there are different types of testing and they accomplish different goals. There are lots of things that unit testing is simply a poor or incomplete solution for, which can lead one to think that unit testing is a waste of time. However, I view it as complementary to integration testing and functional testing.
2. Don't stress out about coverage. There is a time and place to think about code coverage, but if you worry about that right now, you will waste time unit testing accessors and mutating, which provides no business value.
3. Don't think about testing classes or methods, but think about testing behaviors. What does this class do? What does it not do? (This is part of why the term BDD is used these days)
4. Test interesting behaviors. What qualifies as interesting is up to you. Many experts argue that a bug is a missing test. Therefore, anything you feel could reasonably break is worth testing. Would you write a test for i++? Probably not. Would you write a test for incrementI()? Maybe. If it's a calculator app, I probably would. If it's a web app, then I wouldn't sweat something that simple.
5. Tests are not just about making sure your code works. They serve as an executable specification. Meaning, if somebody needs to understand the requirements of a class, they can look at the test.
6. It is ok for a test to assume that a dependency works. If that worries you, then maybe you should unit test the dependency as well. But keep it separate. For example, one could argue that testing the data processing without a real file is bad because you don't know if the file parsing works. But if you have a unit test which verifies that file parsing works, and a unit test that verifies the processing of the file's data works, then clearly both work. Granted, this does not prove that they work together. But that's ok. That is integration testing.
7. Unit tests should be repeatable and isolated. A unit test that makes a web service call, then writes to a database, and then prints some result to the console is not a unit test. It's a test, sure, but not a unit test. Don't get so hung up on testing everything that you write fragile tests which do a ton of work. So if a method does those things, how do you unit test it? This leads to the next topic...
8. Separating out dependencies - Write small units which can be called independently, with dependencies passed in. If you design so that the processing is independent of the source file itself, you can test the processing logic without needing files. For example, instead of the file processing taking a java.io.File object, parsing it, and doing all the work, perhaps it could take a java.io.InputStream, parse it, and do all the work? Or a method which reads the file's contents into an array or other data structure, then passes that into the processing?
9. Mocking out dependencies - ultimately, somebody needs to talk to the external world. Ideally, you will separate your dependencies, but sometimes you reach points where the complexity introduced by doing so is counter-productive. For example, a method which several levels down the call stack needs to talk to the database and separating that out is unreasonable, then mock it out with a fake implementation that returns test data. I've gone as far as creating a MockDatabase class before, which lets me map stored procedure names to hard-coded, in-memory result sets.
10. Realize when you have mocked so much that unit testing is pointless. If you have mocked out everything a method does, then you are left testing an empty method. If that is the case, maybe that method is not interesting to test? Trust that an integration test, automated functional test, or even manual testing will catch any problems with it, and don't worry about unit testing it.

I could go on forever. Hopefully that helps.

Remember, this is not easy. If it was easy, everybody would do it.

Answer 2

It's important to understand TDD as less about the end result - tested code, and more about the journey in getting there. TDD is a design process that encourages all the code qualities I'm sure you have learned about: reusability, separation of concerns, readability, maintainability, etc. Oh, and of course, correctness. Your tests drive you towards writing isolatable, testable code, which in turn drives these qualities. If you only understand TDD as about testing, you are in for a hard time.

Maybe you have already written this program (or maybe you have at least designed it in your head), and you are thinking about how to wrap tests around it. If this is true then start again as you are not driving the code from the tests: you're not doing TDD.

Writing your tests first, then making each pass, might lead you towards the following units:

• A thing that opens a file and exposes a byte stream
• A thing that parses a byte stream as a PDF
• A thing that takes the representation of a PDF and extracts what you're interested in
• A thing that groups by something extracted previously
• A thing that allocates a file location based on group
• A thing that moves a file to a location

(I may not have understood your program correctly, but you get the gist.)

Hopefully you can see that you've broken it down into easily testable units, many of which can be reused, most of which are agnostic of files, or PDFs. Those units that actually deal with real file operations are very simple and are few and far between, so testing them is not difficult; everything else deals with some in-memory representation.

It's not easy to get right at first, but the golden rule is to remember that if your tests becomes complicated, it's probably a clue that your code has become too complicated and is trying to do too much in one unit. It's your cue to take a step back and keep asking questions like these.