We have these things called "executions" that are run at certain times. They have a
delay property that basically says "execute me after
delay microseconds`. Currently we are using Cassandra as a queue, but this is definitely not scalable, and the database is not the best tool for a queue.
I could implement this as a priority queue, but since we expect to receive a large number of messages, I cannot keep this in memory. I was thinking of using Apache's ActiveMQ which supports priority, but I'm wondering if it's worth it to bring in an entire framework. My other option is to implement a priority-queue that is written to disk (using
MappedByteBuffer). The second option is more work, but more directly addresses our use-case. I don't mind using ActiveMQ and it would most certainly be easier, but I'm just wondering if it is worth it to bring in an entire framework when I'm only going to be using a small part of it.
Some more context:
We have these scripts that pull in data from different sources. The script returns the data, along with an "execution" object that says "run me again after
delay microseconds" (basically polling for new data). Currently this execution is written to Cassandra, and a thread periodically polls and picks these up and then calls the appropriate script.
As I read more into message queues, I'm realizing that it may not even work. ActiveMQ has a fixed list of priorities, whereas I want a priority queue that supports arbitrary priority-values.
I guess it's more or less a scheduler. Each execution is "scheduled" to be run after
delay microseconds. Would it be more appropriate to use a scheduling framework here? Is there one that supports priorities?