Using short-lived REST requests to obtain messages from a message queue for clients

Question

I would like to put together a system where clients obtain messages from a REST API, but with the producer being able to use the semantics of a message queue: e.g. ordering of messages, a destination mailbox, etc.

I figure I can have one message queue per client.

When a client connects to one of N stateless webserver/REST endpoints to GET new messages, any new messages gets dequeued from the client's mailbox on the message queue, packaged up and sent to the client as a response to the GET.

I think there are some problems with this approach, though.

Seems each REST request would make a new connection to the message queue (or possibly there is some pool of connections - but mailboxes would have to be efficiently switchable in the connection)

Furthermore, I am not sure how to handle the case when a client gets a list of messages as a response to a GET, dies halfway through processing them, and has to re-request messages it has already been given. The queue would have to have some level of persistence, and some ability to rewind.

My questions are: am I going down the right path here? What can I do about these issues and am I missing any other pitfalls? Or maybe there is a completely different way to do this that might be more optimal?

Answer 1

It is not because you use the semantics of a queue that you need an actual queue to back your REST service. It is very possible to create a service that uses queue-like semantics, but that is backed by a database.

In that case, you would not have a new connection per REST request but would take advantage of connection pooling. Using a database also means messages are persistent so a degree of rewinding is possible. As for dying receivers, a request to get messages could send a list of messages to the client in one request and mark these as sent in the database. The receiving application could then acknowledge receipt of a message by individual REST calls. A job running intermittently could re-mark messages to be sent to the client if no acknowledgement was received after an amount of time.