I have a service that runs queries on a database and gets (potentially) large number of rows. I also have services that can export a given set of rows to csv, xlsx or pdf etc. I have separate export services to leverage language capabilities, for example, java has a better xlsx exporting library.
What is the best architecture to stream my query results to these export services in a horizontally scalable way?
I tried using message queues like
rabbitmq however the problem is the message size. If I send message in chunks/rows, there is no guarantee that the same worker will receive all the rows due to round robin delivery.
Any tips/ideas are greatly appreciated