3

We have these things called "executions" that are run at certain times. They have a delay property that basically says "execute me after delay microseconds`. Currently we are using Cassandra as a queue, but this is definitely not scalable, and the database is not the best tool for a queue.

I could implement this as a priority queue, but since we expect to receive a large number of messages, I cannot keep this in memory. I was thinking of using Apache's ActiveMQ which supports priority, but I'm wondering if it's worth it to bring in an entire framework. My other option is to implement a priority-queue that is written to disk (using MappedByteBuffer). The second option is more work, but more directly addresses our use-case. I don't mind using ActiveMQ and it would most certainly be easier, but I'm just wondering if it is worth it to bring in an entire framework when I'm only going to be using a small part of it.

Some more context:

We have these scripts that pull in data from different sources. The script returns the data, along with an "execution" object that says "run me again after delay microseconds" (basically polling for new data). Currently this execution is written to Cassandra, and a thread periodically polls and picks these up and then calls the appropriate script.

As I read more into message queues, I'm realizing that it may not even work. ActiveMQ has a fixed list of priorities, whereas I want a priority queue that supports arbitrary priority-values.

I guess it's more or less a scheduler. Each execution is "scheduled" to be run after delay microseconds. Would it be more appropriate to use a scheduling framework here? Is there one that supports priorities?

Improve this question
4
  • it sounds like you're talking about a task scheduler, and strange one at that; please provide more context
    – Steven A. Lowe
    May 29, 2015 at 19:53
  • @StevenA.Lowe Yeah, it's more of a scheduler I guess. I got hung up on the implementation. I will provide some more context.
    – Vivin Paliath
    May 29, 2015 at 19:53
  • Look into the concept of timers. This is really not the right thing for a queue, operating systems have for a long time had the ability to schedule events and most languages have some form of abstraction over this ability often referred to as a "Timer", though sometimes having other names. The concept is you set the timer to go off in a period, and you direct the timer as to what it will execute - or what signal it will flip when the alotted time has passed. The OS will handle it from there signalling to your process to execute the code at the appropriate time.
    – Jimmy Hoffa
    May 29, 2015 at 21:18
  • I do need to store these events as they come in, and order them by priority. Then I will have them process them by order of priority. AFAIK you can use a priority queue for processes. I wrote a small preemptive multitasking OS in college for the Motorola 68k where we used a priority queue similar to this, to figure out which preempted tasks to resume, based on priority. Are you saying that as the events come in, I simply set up a timer to have them processed at some time? The problem is scale here since I can have tens or thousands of events.
    – Vivin Paliath
    May 29, 2015 at 21:21

2 Answers 2

Reset to default
1

Think about what you really have to keep in memory in order to execute the jobs; it may be as simple as just a job ID, next-execution-time, and periodic-repeat-time. In the simplest (and possibly oversimplified) case, everything can stay in the database and the execution 'engine' only has to know the next job to execute.

Profile it, consider likely and peak loads, then add work agents and dispatchers and queues (oh my!) as needed. Or something completely different ;)

Improve this answer
1

As I understand you need to:

  1. run the script
  2. get the data and schedule next script execution after delay with some priority

I would implement step 2. using some scheduling framework, like Quartz. Quartz provides ability to prioritize executions in case they are triggered at the same time:

Trigger trigger = newTrigger() 
    .withIdentity(triggerKey("myTrigger", "myTriggerGroup"))
    .startAt(futureDate(delay, SECONDS))
    .withPriority(priority)
    .build();

Full example can be found in Quartz tutorial.

This approach does not require any additional database tables. If you need persistence you can use JDBC JobStore configuration to store scheduling data in database.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.