0

In a regular WSGI service provider implementation there is a blocking HTTP server that services clients. When a client, for example, creates an invoice, the server simply makes a CRUD operation that creates rows in necessary tables. Then there is a worker - typically a shell script that uses the same set of models the server framework (MVC) uses. This worker (applied to the example), say, runs in cron every n seconds and checks the bank account for any new transactions. When a new one is found (and it matches the account of the previously created invoice), it does the CRUD again to update the invoice rows previously created, etc..

Now, when we are talking about async HTTP server implementation (for example Python Tornado), how do we approach the 'workers' part? Because there still must be some process that checks the bank account for any new transactions. Do we just run one IOLoop, which inside runs the actual HTTP server app, as well as all the necessary 'workers', which are simply several separate apps, that, again, use the same model set from the main server app? I mean, we can potentially run the workers as completely separte processes from cron, but then async doesn't make much sense, does it? Or should the 'workers' be implemented comletely differently (eliminating the 'worker' concept?) and I am not grasping all the async stuff correctly? Thanks!

1

It looks like your system has a producer-consumer scenario, where one part produces data/message for other part to consume it.

If it is so, you can well utilize a Message Broker like MSMQ or RabbitMQ. Since they allow asynchronous communication between two applications.

Whether it is feasible for your system or not would depend upon it's complexity, availability of resources etc.

  • Yes, the system is of a producer-consumer type. We are going to be using a message broker. The 'producer' here is actually a separate system that pushes messages to ours. I am just trying to ask if it is reasonable to create a consumer using an async HTTP server and alongside in the IOLoop run the processing worker as well? Look at this: server (Tornado) waits for new requests, when request is received, the server puts it in a queue (RabbitMQ), then some time on the loop run the worker processes the request. Or instead the worker should be an independent process/app? – lime Jul 14 '16 at 11:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.