Exception handling philosophy when dealing with batches of input

Question

I am handling a file with 1 million records and my customer creates this file from his code. I might receive hundreds of these files. I have to read these records and ingest them into a database. It is possible that some of the records might be incorrectly formatted and lead to parse exceptions. What should be the exception handling philosophy here?

1. I could handle the incorrectly formatted records and store them in a queue while processing all the other correct records in a file. This would require me to write extra code but once the customer fixes this error in his code the error handling code in my application will not be used ever and I would have wasted time writing and maintaining error handling code that might never be used or used sparingly.
2. I could fail the entire file by letting the RuntimeException percolate upwards and let the customer correct his code and then re-submit the file again. This would keep my processing simpler but would mean that I am not processing the correct records even though I could handle it in the first pass.

Answer 1

All options are viable. It all depends on your scenario. Sometimes the best option is clear cut, and in other cases it leads to choosing the lesser of two evils.

I'll use a real world example that I dealt with recently. Our customer would import an excel file containing a year's schedule, and we needed to parse all data and add it to the database. Similar to you, we pondered what to do if one of the entries was corrupt. We couldn't decide on whether to split out the errors from the successes, or fail the entire import. Both cases had negatives:

• If we fail the entire import, then an entire company's yearly planning could not continue until they fixed one problem in one entry.
• If we still handle the successes and report the error, the customer will fix the error and will then import the whole file again (note: when reimporting existing data, it was a full drop and recreate, so we could not easily skip existing entries). This would mean that processing the entries the first time was futile as they would be deleted during the next import anyway. Once you knew the import was not fully done, you knew the rest of the work would be in vain too. By immediately failing the import, we would improve the user experience by not having them wait on a long and inconsequential import process before realizing that something went wrong.

In the end, we went to the customer, explained the two options (and the consequences of both), and let them decide. This decision was not something that affected the developers, but it would massively impact the business who would be using the application, and therefore we let them decide which of the evils they were more prepared to deal with.

Answer 2

How independent are the records? It might violate a business rule to add one record and not another. Until you have an answer to that question the whole file needs to be rejected.

Regardless of how that pans out you need to produce a debug friendly error message that details which records failed.

Whatever you do leave the system in a well defined state or roll over and die. Otherwise you've no idea what happens next.

Answer 3

It is possible that some of the records might be incorrectly formatted and lead to parse exceptions. What should be the exception handling philosophy here?

The implementation philosophy can only be defined once you know what your customer wants.

Do they want you to fail the whole process if there's an error or do they want all the valid records added? Do they want you to try and recover common errors, do they want a report on the errors, do they want to fix them themselves and resubmit? Do they want to resubmit just the fixed records or all of them and then want you to only process the previously failed ones?

Once you know what is required, then you can start thinking about your implementation. Before that, all we know is that jumping straight into asking what exception strategy to use is absolutely not the right philosophy.