4

I have what I believe is a fairly common situation these days: I have a webapp that scales horizontally across multiple machines in order to serve greater load. This particular application is written in Java.

On this same webapp, I find that I have a need to run a background process in this webapp periodically. However, this process executes requests against another service, and I don't believe it would be safe to have multiple invocations of this process going at once.

It would be easiest to just get a new machine that runs these background processes by itself. However, company policies make it difficult to get new machines, so I'm trying to investigate other options in case this isn't possible. This means I'm trying to figure out how to make just one machine at a time run this background process at once. I've thought about some of the following ways:

  • Figure out how to make one machine the "leader" and be the only one to execute these periodic background tasks. This is complicated by the fact that I'm running on an AWS auto-scale group, so this leader could be killed at any time, so I'm stuck with needing to do leader election on my own.
  • Establishing a mutex of some kind that each instance must grab before it executes. I've thought about using an in-memory data grid like Hazelcast for this, but that seems a bit heavyweight to set up and use just for this.

What are some good ways of achieving my goal of running this background process in my webapp and making sure it only runs once?

Improve this question
2
  • Why not run this background process on a dedicated machine outside your auto-scale group?
    – Dan Pichelman
    Commented Nov 23, 2015 at 17:55
  • Ah good question, I forgot to answer that. That would certainly be easiest for me. Due to company policies it's difficult to get new machines. That's a possibility at this point, but I'd like to investigate my options in case that request is rejected.
    – dsw88
    Commented Nov 23, 2015 at 17:59

4 Answers 4

Reset to default
3

Can you add another piece of software to your architecture? If so, I would recommend you to use some sort of queue, let's say RabbitMQ, for that matter.

In this scenario, you won't have to worry about managing which process is running at a given moment (or will run next). They stay there waiting for a job requisition to arrive via queue. One of them will "take the order" to start the request to the external service. I see this as an out of box implementation for your idea of a "mutex".

What you have to figure is how to enqueue this "now we need a new request for an external service" trigger.

Improve this answer
2

It looks like you need a scheduler. These typically allow periodical starting of jobs, and will also allow configuration to make sure the job is only started once. We use the Quartz scheduler, but there are many more options. Also, if you run on a recent JEE application server, these might provide this functionality natively, see JSR 352 for the description of the JEE 7 standard. This might also provide the required functionality.

Improve this answer
2
  • This, or good ol' fashioned CRON jobs or scheduled tasks in Windows. As long as you don't schedule two instances at the same time you're good. You could also use one of the web servers as a background job server. One that is least often used by the load balancer.
    – Greg Burghardt
    Commented Nov 24, 2015 at 17:55
  • Yes, I agree. But the advantage of a scheduler is that it will do that automatically for you, and also will fail-over if an instance fails. Also, the question had a java/web-applications tag, so a java solution seemed appropriate.
    – Geert-Jan Hut
    Commented Nov 26, 2015 at 12:13
1

If your distributed webapp uses a shared database, you can implement your suggested mutext by using a database table row and locking.

But I think the best and sound solution would be to use a dedicated machine and service that runs this background process on it's own dedicated machine. If cost is the issue then you can minimize the cost by only having the machine up and running periodically when the request is going to be made.

Improve this answer
1

There are multiple possible solutions/technologies:

  • If you have a shared database, just use that to create a mutex
  • You can use JGroups, which is a lightweight communication/coordination library. Can be easily embedded and has primitives for distributed locking.
  • You can use Zookeeper. Which is a distributed, but ACID/transactional hierarchical "database" of sorts. Locking is supported. Can be embedded (sort-of), but requires filesystem to store its database.

More complicated/involved/heavyweight solutions:

  • Full messaging, like RabbitMQ, Kafka, etc. (Kafka actually includes a Zookeeper, so it doesn't really makes sense unless you need messaging anyway)
  • JEE Container's native Scheduling. This is tricky, and I personally had endless problems with this, especially in distributed environments. This would require a shared database anyway for the implementations I know.

That's just from the top of my head, there's probably a lot more options..

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.