Architecting Python application consisting of many small scripts

Question

I am building an application which, at the moment, consists of many small Python scripts.

Each Python script processes items from one Amazon SQS queue. Emails come into an initial queue and are processed by a script, and typically the script will do a small unit of processing (for example, parse email and store some database fields), then an item will be placed on the next queue for further processing, until eventually the email has finished going through the various scripts and queues.

What I like about this approach is that it is very loosely coupled.

However, I'm not sure how I should implement live. Should I make each script a daemon which is constantly polling its inbound queue for things to do? Or should there be some overarching orchestration program or process? Or maybe I should not have lots of small Python scripts but one large application?

Specific questions: How should I run each of these scripts - as a daemon with some sort or restart monitor to restart them in case they stop for any reason? If yes, should I have some program which orchestrates this?

Or is the idea of many small script not a good one, would it make more sense to have a larger python program which contains all the functionality and does all the queue polling and execution of functionality for each queue? What is the current preferred approach to daemonising Python scripts?

Broadly I would welcome any comments or opinions on any aspect of this.

thanks

Answer 1

If this application needs to stay running for a long time then build in resilience but kill it at random times (the monitor too - but less frequently), and have a monitor restart it. That way you are constantly exercising the monitor and restart capability of the system so when some true system failure shafts your system it will be more likely to recover smoothly.

Answer 2

I think this all depends much on your own goals and constraints. Does it run fast enough this way? Is it flexible enough for your use cases? Is it too brittle/Does it fail often? Does it seem well organized and easy to update/maintain/enhance?

If it works, do it. If it fails to meet your requirements, change it. You may need to test it "live" to decide whether it is acceptable.

A single program consisting of a framework for integrating the various scripts as modules would probably be ideal. But is the work required for a Cathedral worth the gain? Could you settle for a Bazaar?

Answer 3

An alternative to queue processing is to utilise a noSQL schema with a status field - eg. 0-New, 1-pending, 2-processed OK, 3-Processed error and custom fields for whatever processing tasks you need.

Within your schema you could put processing results or useful error data. The important benefit of this approach is transparency. At any moment you can query the pipeline status.

It is important with this approach to use an atomic update (in Mongo findAndUpdate) - so that no two processes will retrieve the same task.

Answer 4

I'd propose turning scripts into standalone, atomic services (yeah, I know, services SHOULD be atomic, whatever). Then, you can find some ESB and glue it all together. Most popular (I think, "citation needed") ESB for python is zato.io, but I haven't tried it. Instead, I've seen MuleESB (Java solution), and it works like charm.

"Java solution for python scripts?" you'd ask. Yeah, that will work. Services are language-agnostic, they are little black boxes, that do only one thing - like you wanted. You glue them together with ESB and some queue system (ZeroMQ? ActiveMQ? your choice), and there is no difference, if you're gonna write services in python2, python3, Java, C(++), Smalltalk or even damn Brainfuck.