Integrate an external rest api with my web application

Question

I have a service running on AWS exposed via a rest API. The service processes files.

Clients of the service post a file to the api and are returned a Job ID.

Clients can then request the status of the job, if it is complete the job result is returned as JSON.

The API essentially consists of a simple gateway service that accepts the job requests and puts them on an SQS queue. Then there are a pool of ec2 workers that process jobs and update a database in RDS with their results.

The actual time taken to process a file can be between 20 seconds and 2 minutes depending on the file size, however depending on the queue size and size of my scaling group it can take several minutes from when a client submits a job and to when it is complete.

I am now building a separate web application that allows humans submit jobs to this service. The user will upload a file to the web application, and will then see some sort of 'Processing...' screen and then when the job is complete will see the results.

I am unsure as to the best way to handle how the web application knows when a job is complete.

At the moment this is my approach:

1) When a user submits a file, my web application posts the file to the processing service and is returned a Job id. That Job Id is stored in my local DB with a state of 'processing'. That Job Id is added to a local 'Jobs in progress' queue in my web application

2) The web application returns the job Id to the browser and the browser polls the web application checking the state of the Job Id in the database

3) A separate thread pulls items from the local 'Jobs in progress' queue and for each one polls the API service checking if the job is complete. This thread pulls say 50 items from the queue and a time, and checks the status of each job against the API every 10 seconds. If the job is complete its status is updated in the local database along with the job results.

I know this will 'work' but it seems cumbersome and I wonder how it will scale, and if I am deploying this in a load balanced environment (say on AWS) how will it cope with the potential failure of one of my web app nodes.

It also seems to me that this is a common use case, and perhaps there are some better patterns.

The web application is being built using Spring.

Answer 1

You should consider using server-sent events. Here are some places to learn more about it:

• MDN Using server-sent events article
• Server-Sent Events chapter in the online book High Performance Browser Networking
• HTML5Rocks Stream Updates with Server-Sent Events article
• HTML5Doctor Server-Sent Events article

Server-sent events are natively supported in Chrome, Opera, Firefox, and Safari and there are polyfills you can use to make server-sent events work in Edge/IE as well, and even a jQuery plugin.

Here’s a very simple example of some client-side code:

var source = new EventSource("/server-side-app-that-gives-progress-updates")
source.onmessage = function(e) {
  console.log(e.data);
}

On the server side, your /server-side-app-that-gives-progress-updates app just sends back plain-text data formatted like this:

data: some data\n
data: more data\n\n

It starts streaming that by sending a response with a Content-Type: text/event-stream header. The client code above will see it as a single e.data message with "some data\more data".

When the server app has an update, it just writes another message of that form to the response.

If you want to send the updates as a series of JSON messages instead, your server app writes:

data: {"foo": "bar", "hoge": "moge"}\n\n

And then your client code can do this:

source.onmessage = function(e) {
  var jsonObject = JSON.parse(e.data);
  console.log(jsonObject.foo);
}

Of course there’s more to it you can learn about at the places cited above. But that’s it in a nutshell.

Using server-sent events would eliminate the need for you client code to do any polling. The key use case for it is one-way communication of updates from the server to the client (in contrast to web sockets, which would be the thing to use if you needed to also do a lot of real-time updates in the other direction, from the client to the server).