How to design error monitoring for batch jobs with multiple and diverse steps?

Question

I'm working on a software to integrate an e-commerce store and the company ERP. All the integration steps are run as a unique batch job, every 5 minutes. Things are working under production, but I'm struggling with how to better design the subsystem that alert us (developers and IT folks) that an integration error is occurring.

Essentially, the software integrates all the new customers, the new invoices, stock, products, and so on. To simplify things, the process was developed to be run as a serial sequence. Example: integrate first the customers and after the invoices, so we can't have invoices without customer data. When something fails (our e-commerce provider try to integrate some invalid data on a specific customer or it is simply down), the entire process fails and it signals to Nagios (a monitoring and alerting system) alert us.

I'm changing the design so a specific problem with a single unit don't stop the process, but the problem is the design of alerting that the integration of that specific unit failed, and make not necessary fiddling with logs to finding what is happening.

I've thought about some ways, but I would like to know if there is some better way to tackle this:

• Send an e-mail to IT with a simplified description of the problem, with some data (example: invoice number), so they can open an issue ticket with the e-commerce provider. The problem is that the job is tried at every batch processing, so we should have a way to send only one e-mail per problem.

• Send the problem to a centralized system (to be developed) via API, and show a web page listing the current problems. There should be a way to, again, deal with duplicated errors, and remove the resolved errors from the listing after they are resolved and the units integrated.

I am really stuck at this design, and would like to know if there is any other better way to deal with those batch jobs data inconsistencies, even if it demands developing some "infra structure" to make it easier for us.

Answer 1

The problem is the design of alerting that the integration of that specific unit failed.

Take a step back from the technical solution and think about it from a business perspective first.

What types of errors are expected to happen? Go through each integration step and list the things that can go wrong. Group these into different severity levels. For example, these may be minor (e.g. the system reconnected after a short disconnect), moderate (e.g. a record was dumped because it was invalid) or major (e.g. the system is down).

You may need some intelligence here. For example, lower severity (e.g. data conversion) issues may indicate a more severe error (e.g. a whole input file being invalid).

Who needs to respond? Different errors may be fixed by different people at different times of the day. You will also need backups in case the designated person is unavailable (e.g. sick) or busy (e.g. dealing with another issue).

The answers to these questions determine the best medium. Less important errors can probably be stored for later review. Data conversion errors are probably best collated and addressed in bulk. Serious errors need to be communicated to users urgently but you may want to use a different message over night (e.g. SMS) than during the day (e.g. E-mail or Slack message).

You do not want lower severity issues flooding users. This just trains them to ignore errors. It makes it harder to find the root cause.

Also consider other requirements. Do you have service level agreements? The number of failures may contribute to this. Do management what reports the types of failures and time taken to fix them? Do people get paid extra to deal with issues out of hours?

I've thought about some ways, but I would like to know if there is some better way to tackle this.

The good news is many systems out there that do exactly this sort to thing already. For example, most syslog servers have a rules engine that can take the various messages from different sources (each integration step) then run a script if conditions are met (e.g. send an E-mail or SMS). This separates how systems report errors (e.g. via syslog), the rules that determine whether actions are taken (using the rules engine) and what those actions are (the scripts).