I'm working on a software to integrate an e-commerce store and the company ERP. All the integration steps are run as a unique batch job, every 5 minutes. Things are working under production, but I'm struggling with how to better design the subsystem that alert us (developers and IT folks) that an integration error is occurring.
Essentially, the software integrates all the new customers, the new invoices, stock, products, and so on. To simplify things, the process was developed to be run as a serial sequence. Example: integrate first the customers and after the invoices, so we can't have invoices without customer data. When something fails (our e-commerce provider try to integrate some invalid data on a specific customer or it is simply down), the entire process fails and it signals to Nagios (a monitoring and alerting system) alert us.
I'm changing the design so a specific problem with a single unit don't stop the process, but the problem is the design of alerting that the integration of that specific unit failed, and make not necessary fiddling with logs to finding what is happening.
I've thought about some ways, but I would like to know if there is some better way to tackle this:
Send an e-mail to IT with a simplified description of the problem, with some data (example: invoice number), so they can open an issue ticket with the e-commerce provider. The problem is that the job is tried at every batch processing, so we should have a way to send only one e-mail per problem.
Send the problem to a centralized system (to be developed) via API, and show a web page listing the current problems. There should be a way to, again, deal with duplicated errors, and remove the resolved errors from the listing after they are resolved and the units integrated.
I am really stuck at this design, and would like to know if there is any other better way to deal with those batch jobs data inconsistencies, even if it demands developing some "infra structure" to make it easier for us.