We have REST endpoint that that works like this

  1. Retrieve payload from request
  2. Generated unique id for payload and order_status and enrichment_status to PENDING
  3. Store payload in Redis
  4. Perform request to Salesforce to create order and retrieve the result
  5. Depending on the result, update the order_status=Success in payload in Redis
  6. Perform request to enrich order in Salesforce and retrieve the result
  7. Depending on the result, update the enrichment_status=Success payload in Redis

It is done as a single function. The problem is that for now in case of any issue with Salesforce we just throw the error and don't do anything else. So in case if enrichment or order creation fails, we know it only by comparing the amount of entries in Redis with number of entries created in Salesforce (something similar).

Now we want to implement the solution where we want to handle Salesforce errors (by notifying by email and slack) and all requests (whether for creation of order or doing enrichment) should be send every 1 hour again and again until they succeed or something else is fixed.

We don't have that huge load in our requests (though we have a lot of records in Redis) and we use Redis HA for everything persistence related (no even RDBMS). All is written in Python and I can leverage stuff like Kafka (no other message broker) and Celery in our system.

What architectural approach I should go with?

I understand that the simplest way (without queues) what we need is

  • one cron job that would read all orders with order_status=PENDING/FAILED, and another cron job that would read all orders with enrichment_status=PENDING/FAILED and order_status=SUCCESS
  • each job will then send the corresponding request to Salesforce (whether for enrichment or order)
  • if request fails, the job will notify by email and in slack and set the status to FAILED or SUCCESS

However I presume it would not be very effective? Especially considering that I would need to query Redis, which is kinda not designed for that.

Personally I would go with a queue, that would contain the pairs like (unique1:operation). The queue would be processed by some scheduler (every 1 hour) that would read the entries and send them to corresponding topics (like ORDER topic and ENRICHMENT topic). Then the consumers will read the message (it is removed from queue) from the topic and will try to perform the operation. If processing succeeds, then they would update REDIS entry with SUCCESS and done. If it fails they would notify by email and slack and send the message back to the queue with FAILED status.

What approach I should use here? Leverage Event Sourcing (like create topics with events and update state in Redis)? Redis Streams (no Kafka involved)?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.