My team and I are integrating with a 3rd party company and using their API to perform different CRUD operations. Their API isn't always reliable though. Maybe 0.1% of the time an API call just fails with a 500 error, and then when you try again, it works fine. Occasionally, they will have times where the API call fails with a 500 error more than 90% of the time. After ~10 attempts, it will finally work. It usually lasts for about 4 hours and occurs every few months. We experienced this yesterday where 90% of the time the API call would fail.
The 3rd party told us that we should implement retry logic when receiving a 500 error. For idempotent operations, I could see this being useful. However, for non-idempotent operations it seems like it could be dangerous. For example, one API call might be to send an Email. We've noticed at times that the API might return a 500 error, even though the email was sent. If I retry the API call 10 times, I might get 10 500 errors, but the recipient might still get 10 emails. This could be very bad.
Should we actually be adding retry logic for 500 errors that are not idempotent?
My two thoughts on when we should add retry:
- When we are okay with the operation running more than once
- When the operation failing is worse than the operation succeeding 10 times (or whatever the retry limit is).
I've never had to add an retry logic to API calls as getting 500 errors is very slim. Is it reasonable to have to do this?