We have a REST API which calls a third-party REST API to Send Emails. The Third Party API is not super reliable and randomly fails every now and then with a 500.
Our Clients do not want to retry at all and instead requested us to build a retry mechanism for failed emails.
We are using Spring-Retry to implement Retry and Circuit Breaker Pattern where in Fallback method we are storing failed request somewhere (DB/File still an open question).
We have a scheduled job that will run every hour, pick up all the failures where initial retries were exhausted and try to re-send emails.
My question is on if there are any best practices on how do we store the failed request:
- Shall we store the request as is with Body, URL, and Headers in a blob/text in db so it is easier for the Scheduled Service to Resend it,
- Shall we write the failed request to a file somewhere maybe S3 and resend it
- Shall we reconstruct the API request from scratch using all the data passed to us by the client and stored in the database already in different tables (acc numbers, usernames, urls) plus fetching API Keys and reconstruction of URLs.
We are leaning towards option 3, there is more development work involved, but we already have all the data stored and can use it to reconstruct whole request. Is there anything I am missing here or any best practices or design pattern I can leverage?