Options for repeated distributed task

Question

Scenario

My project has a need to periodically scan the database and notify administrators for particular issues. For example:

• Any errors in the logs since last update?
• Do values match expected value from external source?
• New clients?

My nodes (plain old windows services) are currently using Azure Service Bus QueueClient for other distributed tasks, so my original thought was to do something with this same component. However, this proved pretty challenging (see below).

From brainstorming I've come up with the following possible options:

Azure ServiceBus QueueClient

Description: Upon receiving a task, the message handler will enqueue a new message set for X in the future.

For example:

var client = QueueClient.Create("DetectErrors");
var options = /* ... */

client.OnMessage((message) => {
    client.Send(new BrokeredMessage() {
        ScheduledEnqueueTimeUtc = DateTime.Today.AddDays(1)
    });

    if (DetectStuffInDb())
    {
        SendEmail();
    }
}, options)

Issue: How to fire that 1st message? This needs to fire when there is no pending message and only fire a single message (not one per node as might be the case if all nodes start the same time).

Azure QueueClient + WebJob

Description: Use an Azure Webjob to hit an endpoint on my WebApi, which will enqueue a new message. (a slight derivation of this would be to use Azure Automation + Runbook to enqueue a message)

Issue: Seems odd to make so many 'hops' to handle each task. [WebJob] -> [WebApi] -> [Queue] -> [Node]. Additionally, this would make development\testing a little difficult since the WebJob may not have access to a localhost WebApi.

Task.Delay + DB

Description: Use Task.Delay to space out recurring tasks on each distributed node & verify with the DB if another node is currently handling the task. Something like

var id = Cmd("SELECT [Id] FROM [T]");
while (true)
{
    var newId = Guid.NewGuid();
    Cmd("UPDATE [T] SET [Id] = '<newId>' WHERE [Id] = '<id>'");
    id = Cmd("SELECT [Id] FROM [T]");

    if (newId == id)
    {
        if (DetectStuffInDb())
        {
            SendEmail();
        }
    }

    await Task.Delay(x);
}

Issues: Requires a lot of querying of the database by all the nodes.

Question

• Any suggestions to how to best handle this scenario?
• Is there a better way to handle this scenario that doesn't require timed\recurring tasks?

Answer 1

My advice would be to not overcomplicate your program attempting to accomodate task scheduling. When it comes to these things, I generally prefer to hinge on cron or windows task scheduler to deal with the specifics of when. This leaves you the possibility to focus only on the how, and you can disregard having to worry about an interface and all that entails in order to set up a scheduled task.

Just create a simple batch file that performs the request to your server and schedule it to run once a day. I imagine this is the sort of thing that a systems administrator would have to deal with and certainly not your typical user, so you can make your boss happy by getting it up and working in less time and still having all the functionality you potentially require. While it is true that this makes your program operating system dependent, it would take little work to get the same thing to work using cron, and it would seem in your case, this discussion is not even relevant.

Alternatively, you could search for a library that provides this support. In any case, I would strongly encourage you to resist the temptation to code this yourself. This is the type of thing that has been done many times over, and if you attempted to do it yourself, aside from lose time, you might discover some serious issues down the road that, if you're lucky, won't get you fired. Don't take the risk and don't reinvent the wheel. It is my personal idea that the best of us programmers aren't those who can write a better version of an algorithm than you, but rather those who gracefully admit that such things have already been written and must only be pieced together with care.

Please let me know how it turns out. :)