We have these things called "executions" that are run at certain times. They have a delay
property that basically says "execute me after delay
microseconds`. Currently we are using Cassandra as a queue, but this is definitely not scalable, and the database is not the best tool for a queue.
I could implement this as a priority queue, but since we expect to receive a large number of messages, I cannot keep this in memory. I was thinking of using Apache's ActiveMQ which supports priority, but I'm wondering if it's worth it to bring in an entire framework. My other option is to implement a priority-queue that is written to disk (using MappedByteBuffer
). The second option is more work, but more directly addresses our use-case. I don't mind using ActiveMQ and it would most certainly be easier, but I'm just wondering if it is worth it to bring in an entire framework when I'm only going to be using a small part of it.
Some more context:
We have these scripts that pull in data from different sources. The script returns the data, along with an "execution" object that says "run me again after delay
microseconds" (basically polling for new data). Currently this execution is written to Cassandra, and a thread periodically polls and picks these up and then calls the appropriate script.
As I read more into message queues, I'm realizing that it may not even work. ActiveMQ has a fixed list of priorities, whereas I want a priority queue that supports arbitrary priority-values.
I guess it's more or less a scheduler. Each execution is "scheduled" to be run after delay
microseconds. Would it be more appropriate to use a scheduling framework here? Is there one that supports priorities?