I'm thinking about refactoring my content indexing process to queues. Right know updates happen directly when editors change content. It is difficult to do maintance works on the search engine too because we can't easily stop write queries.

The solution I see is to index content via RabbitMQ queues and use batch processing as much as possible to reduce number of queries (prefetch count = 100). However I don't have idea how queue consumers should work because I have about 20 languages and I need to process content from each language separately. It doesn't seem to be a good idea to run 20 consumers to listen every language specific queue. On the other hand I can run one consumer and switch between queues but it may be wasting time sometimes.

  • 1
    Why don't you think one consumer for each language is a good idea? It would probably the easiest solution, and enables simple scaling by putting the indexing workers on separate servers if needed. If the demands are low enough that parallelization is not needed, I don't see why the minimal time overhead incurred by a single worker polling the queues would be an issue. It's probably best to just implement one option, observe its runtime performance, and decide then whether the other option promises a significant improvement. – Hans-Martin Mosner Feb 24 '19 at 7:38
  • Running one consumer per language queue doesn't seem to be a good idea because we are going to launch new languages and I don't think we need this scale of parallelization. On the other hand the single consumer isn't best solution because I must index new content within 2 minutes and I think it doesn't guarantee this requirement. BTW I read about aggregator pattern so I could run separate queue to group messages into one message. Maybe it is a way to go. – deem Feb 24 '19 at 7:47

The main thing you should be designing for here is the order you want tasks to be processed:

If all tasks should be processed strictly in the order they are queued, you want one queue with all messages on it. The language would then be flagged in the message contents. However, to scale effectively you will still want multiple consumers of this queue, and should set your pre-fetch value low, so that one consumer doesn't reserve messages that another consumer would be ready for.

If the important thing is for each language to be processed as soon as possible, use multiple queues, so that a large number of messages for one language won't delay messages for another language. Having a single consumer for each queue will make these fully independent, and you can use a high pre-fetch value as you discussed. However, this limits your scaling options: if a particular language has a lot of messages, its queue will grow, while other consumers are idle; manually tweaking this, adjusting pre-fetch values and number of processes for each language, is a waste of your time, which is far more expensive than a few CPU cycles.

If you build a pool of consumers, each polling multiple queues with a small pre-fetch value, you will get somewhere between the two: messages will be processed roughly in the order they're generated, but with some messages "jumping the queue" when they are pre-fetched. Note that most RabbitMQ connection libraries will implement the polling part internally, so you don't need to worry about how to build that efficiently; you generally just give the library a callback to run depending on what queue it spotted a message on. If you have one particularly busy queue - or, more importantly, one which takes longer to process - you can set up a separate worker pool for that queue, and leave the others together.

| improve this answer | |
  • Thanks for your answer. So do you mean to run multiple consumers with prefetch count > 1 and consume from multiple queues? However I would like to process documents from one language at one moment to execute mass index query later to avoid queries with one or two documents to the search engine. – deem Feb 24 '19 at 11:00
  • @deem Unless you add explicit delays, there is always the possibility that only one message has been added since you last polled the queue, so it's hard to guarantee that however you arrange things. Polling multiple queues from one consumer gives a bit of built-in delay, though: the queue for language B can build up while you're processing a batch from language A. – IMSoP Feb 24 '19 at 11:13
  • in comment to my question I have mentioned an aggregation queue. What do you think of that? It seems to be a good idea to run such queue which will aggregate messages per language and publish to indexing queue when message count exceed 100 messages per language or timeout occur (for example 30 seconds). The indexing queue will receive document batches, not particular document itself. I will be able to run multiple indexing workers with prefetch count = 1 and avoid creating separate queues per language. – deem Feb 24 '19 at 12:40
  • @deem I think I see the logic, but I'm not sure how an extra queue for the aggregated batches helps, other than breaking the code up into smaller roles, which I guess is reasonable. The tricky part is keeping track of the partial batches, and the timeout for each. I suppose you could not start a regular consumer at all, instead periodically querying the length of the queue and fetching a single batch of messages when it reaches a threshold or your timer runs out. – IMSoP Feb 24 '19 at 12:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.