Python reading from stdin while doing other tasks

Question

I am trying to write a system log parser. It will receive messages from the FreeBsd syslog daemon through stdin.

It will use those messages to determine if an ip should be banned or not. The problem I have is that after x seconds the ban should be removed, but if I won't get any input from stdin, my program will just block waiting for it. So in the mean time I can't do anything.

I fixed it using threads, but isn't there a better way to do it?

For example something like this:

while true:
    while <data in stdin>:
        handleData

    doSomeStuff()

So as long nothing comes in from stdin I want to execute doSomeStuff, and if there is data handle it.

Answer 1

This is a perfect example to use multithreading.

Create a 'worker' thread and another thread, which waits for user input. When the 'input' thread receives input, it can add an item to a 'todo' list (with some sort of locking to exclude race conditions).

Then, the worker thread can fetch the topmost element and execute it. In your case it should also add another item for 'unbanning' the IP later, which means each task should have a 'time' tag of some sort.

A reasonable structure to implement this is a red-black tree or a binary heap with sorting by the 'time' value.

Then, your worker thread checks if the time has come to execute the top object in the 'todo' list, executes it and sleeps some reasonable amount of time (say 100 ms).

On the other hand, the input thread waits for input, and when presented with it, adds a task with the current time (which guarantees it would be almost immediately picked up by the worker thread).

Answer 2

Threads are fine, and there's really nothing wrong with using them here. For completeness sake, there's also select(), which allows you to do a blocking check for pending input on a file descriptor, with a timeout. If you use this function in a loop, interspersed with the timed activity, you'll get roughly the same concurrent behavior, but implemented on a single thread. This is the python binding for select() and its friends, btw.