I need to design an application which makes requests to an external API. External API has a hierarchy of entities: User which contains ListOfItemss which contain Items. I need to get data on all Items for all Users. Within a ListOfItems requests per Item can be concurrent/parallel but only one ListOfItems can be requested at a time, due to external API limitations (ListOfItems can only be accessed via a token and a token for only one ListOfItems is issued within a given User). There will be around 1000 Users max.

I need to get this data ideally once an hour but in reality this takes longer because the amount of Users and Items is quite large. I already have the code for the method processUser:

processUser(user) {
  List<ListOfItems> lists = getListsPerUser(user);
  List<Items> items;
  for (list in lists) {
    items = getItemsPerListConcurrently(list);

Currently, I have a cron job which runs immediately after the previous job finished. The job has a thread executor which creates N threads and each thread runs processUser inside a Kubernetes pod. So Users and Items are processed in parallel but ListOfItems are processed serially. I'd like to scale the solution to multiple pods but this requires to make sure that at any given time same ListOfItems is not processed by different pods because the token to access user data can only be used for one ListOfItems at a time (if 2 pods process the same ListOfItems then we have a problem).

Initially, I thought to go with a message queue as a solution and having each consumer process some User because there's already a method processUser and this feature needs to be delivered ASAP. So a message would contain a User id. The problem is that it can take hours to process a User. As far as I know many message queues have some interval to acknowledge that a message was processed, if the interval has passed without acknowledgement then the message may be returned to the broker (Kafka has this I think, AWS SQS definitely has this). This means that a message to process User may be stuck in the queue for hours potentially even a day because all consumers are busy processing other Users. It feels like processUser is a long-running task which message queues may not be suited best for.

There're several directions I'm thinking about:

  1. Use some kind of queue/list of tasks where each task represents a User. Spin up N processUser instances, then each processUser instance pops a User from the queue. The pop operation should be atomic so that multiple processUser instances don't process same User/ListOfItems. Maybe I could use Redis BLPOP which is atomic? This solution would be the quickest to implement in my opinion because processUser code is already written. This is good because the feature needs to be delivered ASAP.
  2. Refactor code to use a message queue with ordering guaranteed like Kafka where each message would contain an Item id and message key = ListOfItems id to prevent more than one consumer processing the same ListOfItems. We already have Kafka infrastructure so just refactor processUser.
  3. Use database synchronization (we use Postgres) so each consumer checks a database table UsersInProgress for which User is available and process it. One option is to lock the whole table to make sure that multiple processUser instances don't grab the same User but I think we can just use a transaction where first we check if a User is being processed, if not then making a write to the table marking the User as being processed. Because writes are atomic we guarantee that only one instance will be able to grab a User.

I was wondering if any of the above options sound good taking into consideration that the feature needs to be delivered quickly or maybe there's another option that I didn't think of.

In case option 1 sounds good is Redis/BLPOP really the best solution or there're better solutions?

In case option 3 sounds good will using a transaction to first check if user is processed then mark it as processed work to prevent having multiple instances processing same user?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.