0

About 10 to 15 records are processed per day with the time interval of 5 minutes between each record.

  1. System A inserts a record in DB and sends id of that record to active mq.
  2. System B Listener receives id from mq and pulls records from DB
  3. System B processes that record and posts to System C using rest api call

If 1,2 & 3 records are posted one by one to mq from System A then sequence of records posted to System C should be same.

Due to outage, System C is down while posting record 2 but it came up while posting record 3. In this case, order is missed and record 2 is not posted at all.

I think below should be correct approach:

  • Maintain status column in DB for each record
  • If System B to C is successful update status as succcess
  • Every time System B to C should check any unsuccessful records are there and process them from id lower to higher
  • Scheduler which runs every 10 minutes can pull all the records which are not success and post to System C from id lower to higher

Please give your suggestion about this approach. Thanks in advance.

Note: This is existing system which is going to expire in 2 years so dont want robust changes.

1
  • negative voter pls provide comment always
    – sunleo
    Nov 29, 2023 at 3:58

1 Answer 1

3

When B is actively polling unsuccessful records from a database to send them to C again, it could also poll for the unprocessed records (maybe it is not even necessary to distinguish between unsuccessful and unprocessed records, except for gathering statistics). With a proper index on the status column, I guess this will unlikely become a bottleneck. So, every 10 minutes

  • let B collect and process all unprocessed records

  • send the records (sorted by ID) to C, but mark only the successfully sent records as "processed"

  • at the next run of the scheduler, records which were not successfully received by C before will be sent again

In case order of processing really matters that much as you wrote, stop processing of B by the scheduler during one time slice as soon as the first transmission to C fails.

The point is, however: this makes the message queue quite superfluous - it is simply not needed here. When looking at numbers like "10 to 15 records a day", it is quite unclear why a database with polling isn't sufficient for this use case, especially when polling the unsuccessfully sent records is needed either.

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.