3

I'm about to propose a new architecture for several pieces of the system I'm working on; it's going to entail getting a lot of number crunching and data translation jobs off of tomcat, and on a more fitting platform and I need a little advice.

First, the way this system is currently setup: There is an AS400 and a PostgreSQL database; some of those tables need to be synchronized at various points in the year. An administration application running on Tomcat6 would have a button, "Synchronize" and the struts action class will spawn a new thread to run the job. The job can take anywhere from 15 seconds to 4 days (this is the maximum time we have experienced so far).

I do not like this design. Tomcat is a web server, not a job processing server.

So, the solution I have in mind is to take one of our unused servers and put Jboss 7 on it. then have a simple interface on that job processing server (e.g. web, JMS) that would listen for triggers. when someone clicks on the "synchronize" button on the web page, the job server is sent the signal telling it what job to fire, what parameters are included and to start processing the job.

Years ago I designed a similar concept in job processing where I had a JMS queue that would listen for jobs to come in. when a certain task was added to the queue the system would fire off and run that task. But on that system the jobs would never take more than 5 or 6 seconds, not several days so I'm not sure that JMS Queues would be the best fit here. I looked at Quartz but thats more of a recurring task scheduling system. Although I know I can code around that and make it work, I'm still wondering if thats the best tech for the problem.

Do you have any suggestion?

Improve this question
7
  • What about Beanstalkd? Or, what torquebox.org uses, JBoss HornetQ?
    – Dalibor Filus
    Dec 13, 2011 at 0:36
  • Long number crunching jobs are done in parallel these days. Have you looked into things like that?
    – Job
    Dec 13, 2011 at 0:39
  • 3
    Advice #1: Please learn to use the shift key. It's hard to read things where the capital letters are all omitted. The question might be interesting, but the persistent spelling errors make it slow going.
    – S.Lott
    Dec 13, 2011 at 0:43
  • Advise #1a: its is not the same as it's. You wrote a wall of text but couldn't bother with the apostrophes? Or capitalizing your Is?
    – yannis
    Dec 13, 2011 at 1:40
  • wow, hornetq looks really nice. has anyone ever used it to run jobs that can possibly take days?
    – scphantm
    Dec 13, 2011 at 13:44

3 Answers 3

Reset to default
2

At a previous job, we used what I jokingly called "redundant array of inexpensive computers" {1} to distribute a number of jobs. Some took seconds to run, others took more than a day (quarterly reports). All were designed to be capable of being restarted if they crashed. Some jobs ran on a schedule, some were run only on demand. The schedule was sophisticated enough to include things as simple as "run daily" or "run monthly" or as complicated as "run monthly, except for the last week of the quarter run daily". Any of the regularly scheduled tasks could be run "now" (on demand).

Because the array {2} lacked displays, the "scoreboard" of who was running what job was read from a database table. In order to determine if a job crashed or hung, I took a concept from embedded computers called a "watch dog". In this case, a running job would periodically update a status column with a 0. The monitoring software would periodically increment that status column, and if the number got above a threshold in a certain period of time, would send a message to a person to basically reboot machine X. The idea was to put this "heart beat" at the end of a loop, but not the innermost loop. I planned to have the "pulse" go off about every minute with the checking circuit checking every 5 or 10 minutes.

Notes:
1 - Folks might now call this a "grid" or "cluster", but they were a pile of Pentium 75 and 100 that hadn't been depreciated, so I could use them.
2 - Well, a 1 dimensional array, as they stacked reasonably well and were worthless enough that if they fell over and broke, no one would miss them. They hadn't been fully depreciated yet, so technically could not be discarded, but were slow enough that they were useless as desktop computers.

Improve this answer
2
  • Your redundant array of inexpensive computers is what fuels Google's data facilities, you're in good company.
    – Patrick Hughes
    Jan 14, 2012 at 1:49
  • I don't even know if I think this approach is brilliant or insane, but it reminds me of things I've come up with in the past, so +1. Maybe it's both. :-)
    – Jason Lewis
    Jan 14, 2012 at 2:00
1

You can run Java programs as batch jobs natively on the AS/400 and there is a Java JDBC driver for Postgres.

RUNJVA is the native command, and QSH provides an AIX emulator if you like a Unix environment better.

Note: If you want a more generic Java'ish solution then consider a Continuous Integration engine which is designed to run jobs in a controlled fashion over a very long time either triggered or scheduled, and to avoid having multiple instances of the same job. We use Jenkins for this - it should be possible to make it run what you need too.

Improve this answer
2
  • yea, my 400 guys brought that up. problem is in the next generation of these systems, the client wants to phase out the 400 completely. we will most likely still use ISeries hardware because its already bought and paid for, but no 400 OS
    – scphantm
    Dec 14, 2011 at 22:13
  • If you write it in pure Java using the jt400 toolbox, you can migrate it to another box at that time.
    – user1249
    Dec 14, 2011 at 22:33
0

Me and my team had to wrote a solution regarding a massive XML upload processing and inclusion in database. We used Spring Batch to create a very fast and robust solution capable of processing hundreds of gigabytes in just a few hours. If you use Spring in your development and because of its POJO-based approach, it should not be hard to use it for your task IMHO.

If you want to know more about it, please take a look at http://static.springsource.org/spring-batch/

Improve this answer
1
  • wow, very nice. i didn't know about this, i will certainly take a close look at it. thanks
    – scphantm
    Dec 14, 2011 at 22:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.