2

We have server that runs data processing tasks, using Python, Django and Celery, on Linux. Once a batch of data comes in, a number of Celery tasks are scheduled and they run, some of them in parallel, some depend on the results of earlier tasks, this all works.

Now we have to implement a new task -- a proprietary Windows-only program needs to be called on some of our data files, resulting in some more files. Only one instance of this program can run at the same time (basically we have to rewrite its config.ini before we can start it each time, and there is only one such .ini). It must run on Windows, can't run under Wine.

Sharing the files is easy, there is a network mount that both sides can talk to.

But I'm not sure what the best way is to start the task from the Linux server, and know that it has finished, that would work nicely with the way we currently do the other tasks.

One option might be to give the Windows machine a tiny webserver, let it listen to a request so it can start up once it receives something, have its use its own Celery or similar, and let it call another URL on the Linux task server once it has finished.

But that seems a bit clunky, do I have better options?

2
  • What is the time that this Windows program typically takes to process one set of files? Is that less than a few seconds or rather in the order of several 10-s of seconds or more? Aug 14, 2015 at 12:08
  • More, I'm not sure exactly how long yet but at least a few minutes, possibly an hour. Aug 14, 2015 at 12:20

2 Answers 2

5

Utilizing a webserver for this purpose is actually a standard approach, it is just a simple form of Service Oriented Architecture. Of course, this term might pretend more than there is actually behind it.

To keep this lightweight, without the need using a fullblown Webserver, you can use a tool like node.js. It is the most simple solution I can think of for your problem, node.js provides all the "glue" you need to make that work.

For example, setting up a webserver with 6 (six!) lines of code is shown at the above entry page. For setting up a REST service with node.js, you will find plenty of tutorials on the Web. Assumed your Windows program is a command line program without GUI (otherwise your scenario does not make much sense to me), you can run it the usual manner with spawn and exec, asynchronously. Moreover, I am pretty sure you can make such a server interact with Celery with just a few lines of code.

3
  • You could also consider some other kind of RPC service, e.g. JSONRPC Aug 14, 2015 at 17:24
  • My main change to your answer would be to use Python on windows instead of Node, only because the rest of their software is written in it. It depends on how much they need to interact with the Windows based software. There's also Edge.js for node, if it's easier to interact via .Net with the app. I'd also mention that a Message Queue could be used for the abstraction as well, which may be cleaner depending on their needs, and if multiple systems will be calling, and there are needs for one request to finish before another starts.
    – Tracker1
    Aug 15, 2015 at 6:49
  • @Tracker1: of course there is more than one way to do it. I did not mention Edge because there is no sign in the question that the Windows program of the OP is a .NET program. And the OP already mentioned the Message queuing system he is using, Celery; I would not introduce a second one on Windows, that would probably complicate things.
    – Doc Brown
    Aug 15, 2015 at 8:01
0

The web service approach recommended by @Doc Brown is probably best, but if you don't want to go that route, there are several options for remote command execution from Linux to Windows: Winexe, Cygwin's OpenSSH, and eventually Microsoft's upcoming SSH support.

Not the answer you're looking for? Browse other questions tagged or ask your own question.