0

I have a service that runs queries on a database and gets (potentially) large number of rows. I also have services that can export a given set of rows to csv, xlsx or pdf etc. I have separate export services to leverage language capabilities, for example, java has a better xlsx exporting library.

What is the best architecture to stream my query results to these export services in a horizontally scalable way?

I tried using message queues like nats or rabbitmq however the problem is the message size. If I send message in chunks/rows, there is no guarantee that the same worker will receive all the rows due to round robin delivery.

Any tips/ideas are greatly appreciated

1

Instead of having services to do the exports to csv,pdf etc perhaps you could create web APIs to handle the exports such that once your query runs it converts the results to json then posts them to api/csv then to api/pdf, then to api/xlsx.

The different web api endpoints could be written in different languages. This way if you need a new export format you would just create a new web API endpoint. If you decide to keep streaming your query results to the APIs as opposed to sending it in whole, then you'd have to configure your apis to listen to streams.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.