Catching Exception and Recalling same function?

Question

I have a function which sends data to a third-party's server using a library they've supplied. The function is very simple. It just takes a list of objects to send, loops through each item in the list, and calls the third-party library's function to upload an object. The problem I'm running into is that my list of objects is fairly big (~22,000 entries, though each entry is only about 240 bytes, so only ~5MB in total), and I always get exceptions part way through uploading the list.

I get a variety of errors, typically Internal Server Errors, but also errors related to missing pieces at the level where their library is working.

A solution I've just written (and tested and had work!) is to catch any exception during the upload process, and then recall my upload function, but pass it only the objects in my list which haven't made it through the upload process yet.

This worked when I tested it, though it caught eleven exceptions before making it all the way through my 22,000 entries. Something about writing code this way makes me feel like an abuser.

I've got a couple of ideas to improve my exception handling:

• Catch only certain exceptions - I've only seen three or four different exceptions come up. If I limit my catch to those same three or four, the program will be more predictable (I won't catch something unrelated like a lost network connection on my end, and try to keep sending my data up)
• Limit the number of times I try recalling my upload function - Supposing their server is down, I could end up in an infinite loop. If I limit it to say 50 or 100 tries, that should hopefully be far more than I need.

Is there a standard best practice for this sort of situation? Is this a misuse of exceptions?

Answer 1

Something about writing code this way makes me feel like an abuser.

I do not understand, why you feel this way. This is how life is (in a distributed system): You have one job and you try to do your best to do it. When the counterpart fails for whatever reason, it's not your fault.

What you should do is doing your job in a sane way, meaning define under what circumstances to repeat your job and how to hande failure.

The most common Pattern to deal with this situation is the Circuit Breaker taken from the book Release it by Michael T. Nygard.

A Circuit breaker acts as a Mediator between your system (inside) and foreign systems (outside). A circuit breaker takes care of retry and controlled failure.

Say you embed a ciruit breaker for your local printer, you would hand every print job to a circuit breaker, which queues a given number of print jobs and sends them one by one to the printer. If the printer doesn't answer for a short period of time, the circuit breaker retries sending the data. If a given limit is reached, the circuit breaker doesn't accept further print jobs unless a given state is re-entered.

Answer 2

As others said, retrying after a (rare) exception is fine. But I would spend some time testing whether a failed upload guarantees an exception; if not you need some way to verify which uploads worked.