Observing progress of a distributed system

Question

For a distributed system, there is a requirement of observing the progress of smaller applications on distributed computers (runtime 5 - 20 minutes).

There is a web fronted, which right now only shows a list of those smaller applications (called jobs), with the state of each of them, like preparing, running, finished

So in web-ui, an administrator can see:

• name
• state
• starting time
• call parameters

from any computer in the network, possibly for the whole system.

Each of the properties is stored in the database, so each state change leads to a call to write to the database. There might be thousends of those jobs at a time.

Description of the distributed system:

Central components, served at one location only:

• Database server (holding results of the jobs to runs statistics, have an overview of jobs run the last three month etc)

• Application server (glassfish, java, runs central server software)

Distributed components, each site has at least one, connected via internet / WAN area: (probably about 20 sites, each has 1..4 Job controllers, each job controller runs about 20 jobs in parallel)

• Job-controller component (windows, c#, wcf, starts and observes small jobs)
• Small applications running tasks, started by Job-Controller (the jobs)

So, for a vague estimation:

20 sites * 4 Job-controllers * 20 jobs = 1600 jobs in parallel

each of which runs from 0 to 100 percent in about 5 minutes on average,
resulting in a progress update each 3 seconds.

giving 533 progress updates per second (over the internet)

Now the customer wants to see something like a progressbar for each of these jobs.

At first, I thought this might lead to a high network traffic and to a vast amount of traffic on the database server.

I do not think that writing progress like 1%, 2%, 3% to the database is a good idea. The runtime of those jobs is not very easy to be estimated good (so it is near enough to a real result), but each job can tell very well what his progress is.

What would be a good architectural approach to observe progress of possibly thousends of those mini-jobs? (Please note that a mixed infrastructure is given. There is the constraint that the system will be built upon that. so: Central glassfish + java and per site Windows + WCF + C#)

Right now I think that each Job-controller could update the progress of all jobs it controls every 10 seconds at once. Would that be an acceptable approach?

Answer 1

This looks like a dashboard. There are a number of dashboard platforms out there that can be configured as to data sources, polling intervals, etc.

Let me suggest a rough design, while challenging some of your assumptions.

Assumption: You state that there is a central database that holds the results of each job. I don't see a requirement that it holds the status of each job while it is running.

Assumption: Detailed progress, at the % level, is not required. It is sufficient to show progress (or lack of progress, with a red flag).

Assumption: 1400 jobs, updating at 15 second intervals each (which is about 5% increments), is 5600 updates per minute, or 93 updates per second.

Assumption: The user interface component can extrapolate the speed of progress from recent updates and provide any smoothing of the dashboard animations.

For your architecture, consider using a distributed messaging framework such as Akka.NET. Each job reports its information and progress to an actor on its host machine. Host machines (I presume there is more than one host machine per site) report progress to a site machine, which forwards the job report to the central server. You may decide to update the database with progress, or not, as you see fit.

On the central server, the web server collects and summarizes the job information. For each job, you have the job identification, start time and parameters, most recent progress reported and progress rate, along with the time stamp. This summary is forwarded to the web page itself (via AJAX queries), where the UI takes each job and displays the progress bar and periodically updates it based on progress and progress rate.

This general approach to the web page design allows you to throttle progress updates as you tune the application, perhaps allowing the web page to focus on one or a few sites at a time for frequent updates. It also off-loads progress bar updates to the client where any kind of animation may be displayed.

Similarly, the use of the Akka.NET framework provides robust, fault tolerant distributed, reactive communication and updates. It will allow you to identify sites or hosts becoming unreachable, which should likely be displayed on the dashboard.