0

I have a segment of customers which may range between 1 and 5000. For each customers, I need to do some processing by making some query to the database using the data from the customer. The query is expensive and is the reason why I plan to use a queue to handle these tasks.

I figured a simple queue with one producer and a consumer won't be able to scale. The server is written in Node JS.

Say I get a list of 500 customers and assume it's an array of unique ids [10, 20, 30, ...500]

Now I loop through the array and pass it in the queue like this:

customers.forEach((id) => {
  queue.enqueue({
    customerId: id
  })
})

Now for each customer, I need to run an expensive query when the customer id is consumed.

Here is my question: By running a query for each customer, isn't that the same as 500 customers performing queries at the same time even without a queue. So, did the queue really offload the tasks?

What type of design do I need to implement in order to solve this issue?

3
  • You might want to clarify what "expensive" means, since it will vary widely depending on context.
    – JonasH
    Jul 26 at 10:02
  • 4
    You're jumping to a solution without explaining the problem. Do you want to run the queries for each customer 1) in sequence 2) in parallel with unconstrained parallelism 3) in parallel with some kind of constrained parallelism 4) something else? Jul 26 at 10:26
  • Are you thinking of a message queue? RabitMQ, Kafka, azure service bus, etc. that is different than a regular queue in a program. It's a totally separate application that just handles adding and removing messages from a queue.
    – Ryathal
    Jul 26 at 13:12

1 Answer 1

2

Queues do not offload tasks. Queues just... queue up tasks. Queue can be useful when you have one part of the system producing a burst of tasks or data and then another part can process the tasks/data more slowly.

If the task is idempotent (running it more than once doesn't help), a queue can also help to reduce redundant executions: if the task is already in the queue then you don't add it again.

But if you actually need to do a query more times in a certain amount of time than a server can actually do, then you need a faster server (or group of servers) or a faster query. Queuing won't help you.

1
  • Thank you for the explanation.
    – William
    Jul 26 at 16:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.