0

I have a system where a user uploads a file to import some users but it's necessary to do some validations and this take some time. I would like to create a work queue with RabbitMQ and the user will upload de file and I'll send to the queue and then process it.

What is the best design to use in this case for my producer and consumer? The service where the user uploads the file is the producer but should I leave the consumer in the same service to process the file or should I create another service to process this queues?

If in the future I need to create new queues and leave the consumers in the same application, it is recommended the application that produces, also consume and process?

It's a microservice application

EDIT

Diagram

User uploads the file to the service and the service publish to the load-file-queue.

The service will have a consumer bind to load-file-queue and process the message of files that arrive. This consumer will load the file into the database and publish the records in another queue (process-records-queue) to process the records loaded.

The service will have another consumer to process the messages of the records loaded and update in the database that processed them.

The load-file-queue is a varying traffic and the process-records-queue the traffic depends on load-file-queue.

Improve this question
2
  • 1
    How is the workload divided? Is there a single queue with lots of traffic or multiple queues with varying traffic? Are your consumers limited to a single queue, so they can potentially experience starvation, while other queues have work to do? You can't come up with "the best design" up front if you don't know your requirements. You can rarely get "the best" even if you do.
    – Kayaman
    Commented Jul 3, 2019 at 7:48
  • I edited the question. But the traffic is varying and the service will receive the file, publish a message into a queue, and the consumer will load de file based on the message and publish the records into a another queue to process them.
    – melpin
    Commented Jul 3, 2019 at 13:13

2 Answers 2

Reset to default
1

Depends on the duration of these long running tasks (validations) and desired client (producer) interaction. If the tasks are not too time consuming and clients would prefer a result of the upload in realtime. You can implement this using Servlet 4.0 (Async servlet) service implementation. This would eliminate the need for MQ message broker and service can directly run those long running tasks in helper threads asynchronously and provide results when they're available.

https://docs.oracle.com/javaee/7/tutorial/servlets012.htm

If you're still wanting to implement it using a message broker, you should implement them as separate services (one producing message, and other processing messages). You would also need some form of API/service to provide processing updates or statuses.

Improve this answer
0

Few thoughts.

Split the producer and the consumer. If you ever have issues in consumer and it's the same as producer you will not be accepting the files from users. The goal of MQs is to decouple the services not couple them together.

The producer should not be aware of the Queue it's pushing the messages to. It should push to an exchange with a routing key. The consumer, when starting up, should create a queue for itself (or do nothing if it exists) and bind it to the producer's exchange with the routing key required. This way, you can create a new consumer in the future without the need to change (and deploy) the producer.

Improve this answer
2
  • If I split the producer and consumer, this consumer application should consume these two queues? Or should I create a consumer application for each queue?
    – melpin
    Commented Jul 3, 2019 at 13:48
  • In your case, I would do it in one because it's working on the same database. You don't want two services connecting to a single database - bad coupling. When you update the database schema, you have to deploy both services.
    – bobek
    Commented Jul 3, 2019 at 13:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.