0

We're designing an application that is supposed to monitor a system for certain events, email interested parties to prompt them to take action when relevant and parse their replies.

The application is comprised of several threads that handle separate parts (one for monitoring the state and sending emails, another for parsing replies and updating the state etc.).

The flow of the application depends on several flags that we want to serialise periodically and should the application fail, on startup it needs to load those serialised flags or if no serialised file is found, to just initialise them to a default.

How can we best achieve this? We've tinkered with different ideas, such as:

  • overriding Python's @property setters and getters to automatically save/load those flags every time they are requested/modified

  • having a separate thread that serialises those flags often enough (such as every minute)

  • have each thread take care of serialising the flags it depends on

What we want to avoid is a case where the system takes some action, sets some flags as a result and then crashes before those modifications are serialised.

We cannot use a database and so we've relied on cPickle as the serialisation protocol.

Another issue that results from the above problem is whether those flags should be kept centrally in an object or distributed across the different components that rely on them.

  • Too much is not clear from your question yet. What are the contents of the flags? When and how do they change? Do different threads use different flags? Do any two threads use the same flag? If they do, can they end up with different values for the flag? Must the set of flags be consistent (point-in-time set of values), or it's fine to save flags while they are changing? – 9000 Dec 9 '15 at 14:22
  • Right, the flags are just boolean flags that indicate if a certain action has taken place. A particular flag is set by only one thread but can be read by the others. So, if an event affecting all users has happened, then the flag for that event is set to True, other threads can read that flag and take action accordingly. – Nobilis Dec 9 '15 at 14:33
1

I'd use a separate 'state saver' thread that receives flag values from worker threads via a queue.

The state saver thread would keep the complete picture of flags for all threads, initially either loaded from storage or sent by worker threads.

When a new message arrives via a queue with a new subset of a worker thread flags, the saver thread updates its consolidated flag state and writes it to a new file. Then it can atomically rename the new file to the canonical file name if you only want one snapshot, or remove an old file if you want to keep several latest snapshots.

All worker threads would write their flag updates to the same queue, possibly as simple {flag_name: value} dicts. The queue will have a reasonably large buffer size, allowing each thread to write their updates 2-3 times, so that a burst of changes, e.g. on stratup, won't make the workers wait too much.

A worker thread should post the flags on every flag change event, most probably via put_nowait. If putting a message fails because the queue is full, the worker thread probably should just log this event and proceed, and you should restart the whole thing with a larger queue size eventually.

The saver thread would mostly sleep in Queue.get(True) calls, maybe with a reasonable timeout to allow it periodically write an 'I'm alive' message to the log or something. It should try to get several messages using get_nowait and until it receives an Empty exception, write the state, and then go to the blocking get again.

BTW you're likely not stuck with pickle serialization; you can e.g. use JSON which is easier to inspect if needed.

  • Hi, sorry for getting back late, thanks for the detailed comment, just a quick question, why is a queue preferable as opposed to just serialising all the flags at once? Will using a queue in the way you've described it mean there's less of a chance of missing an update? – Nobilis Dec 10 '15 at 13:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.