I have a system where a user uploads a file to import some users but it's necessary to do some validations and this take some time. I would like to create a work queue with RabbitMQ and the user will upload de file and I'll send to the queue and then process it.
What is the best design to use in this case for my producer and consumer? The service where the user uploads the file is the producer but should I leave the consumer in the same service to process the file or should I create another service to process this queues?
If in the future I need to create new queues and leave the consumers in the same application, it is recommended the application that produces, also consume and process?
It's a microservice application
User uploads the file to the service and the service publish to the load-file-queue.
The service will have a consumer bind to load-file-queue and process the message of files that arrive. This consumer will load the file into the database and publish the records in another queue (process-records-queue) to process the records loaded.
The service will have another consumer to process the messages of the records loaded and update in the database that processed them.
The load-file-queue is a varying traffic and the process-records-queue the traffic depends on load-file-queue.