Creating a queue for web services

Question

I have a program running that collects metadata from the internets.

Every so often a summary of this data is written up (every 30 min or so) and the summary is posted to a database.

The summary is also sent out to many web service APIs.

I'm trying to avoid the problem of this program locking up when webservice APIs timeout. So I'm thinking of creating a queue to process these webservice requests.

How should I create this queue? With a database of some sort? It could be a program storing queued items in memory, I suppose. Is there some standard process for creating a queue for webservice calls?

Answer 1

This sounds like a perfect use-case for Apache Kafka:

Each partition is an ordered, immutable sequence of messages that is continually appended to—a commit log. The messages in the partitions are each assigned a sequential id number called the offset that uniquely identifies each message within the partition. The Kafka cluster retains all published messages—whether or not they have been consumed—for a configurable period of time. For example if the log retention is set to two days, then for the two days after a message is published it is available for consumption, after which it will be discarded to free up space. Kafka's performance is effectively constant with respect to data size so retaining lots of data is not a problem.

Link to the docs: http://kafka.apache.org/documentation.html#introduction

Answer 2

Not sure what your programming environment is?

On Windows I've used MSMQ and a database table for it.

MSMQ is of course a good candidate because it is programmed to be a queue, and a database table isn't.

But when the application was live, I found a database table was easier to monitor. The default MSMQ tooling was (at that time) somewhat limited and a database table is Always easy to monitor. Besides that you can write any progress in the same database record so you easily can get an overview of the current task running.

But.... when your database gets offline and because of that your task throws an exception, you can't write that status to your 'queue' (being the same database) so in that case you are better of using a 'real' queue.