30

At my company, we have a quagmire of disparate cron jobs (on multiple systems) and manually kicked off processes which keep our business functioning that is a result of years of expedient development and subsequent neglect.

Someday, we will need to come up with a more centralized solution for obvious reasons.

One thought that we have been kicking around is to use our Continuous Integration software (Jenkins) to run these processes, which seems logical.

My question is: Are other companies doing this? Is this a generally accepted practice? Doesn't this conflict with the definition of a CI tool implicit in its name? Are there any other options?

Note: https://wiki.jenkins-ci.org/display/JENKINS/Meet+Jenkins

Jenkins claims that it focuses on "Monitoring executions of externally-run jobs, such as cron jobs and procmail jobs". I'm not sure if this is exactly what I am talking about.

Improve this question
8
  • 2
    Can you clarify the nature of the the various tasks & processes you have in mind?
    – Stephen Gross
    Feb 24, 2012 at 20:03
  • a mix of scripts in various languages, java processes, and linux commands
    – smp7d
    Feb 24, 2012 at 20:51
  • We need more detail. What is the nature of the tasks? What do they do? How are they managed?
    – Stephen Gross
    Feb 26, 2012 at 0:09
  • @StephenGross Gather data from external systems for local storage, send notifications to users based on business rules, check disk usage, delete orphans, and about a thousand other things. They are all managed by cron if they are managed at all at this point. Why do you need these details? You can just assume that they perform business critical functions on a schedule.
    – smp7d
    Feb 27, 2012 at 17:38
  • 2
    The reason I need these details is because to help you with your problem, I need to understand the problem. Although you already know plenty about these tasks/processes, I don't; it's helpful to understand the nature of the tasks to-be-run when evaluating what kind of technical solution works best.
    – Stephen Gross
    Feb 27, 2012 at 18:41

8 Answers 8

Reset to default
18

We've been using Jenkins as a cron drop in for a couple years now, and here are some pros and cons:

Pros

  • If you are managing a large number of processes across dozens of servers and multiple environments, it makes many things easier. You get email alerts out of the box, a common dashboard for everything, a web interface for logs, and an easy way to setup additional nodes to run jobs on. Support teams especially appreciate having this central location for checking problems and re-running jobs.

  • Jenkins plug-in ecosystem is very active and provides a host of additional functionality... I think this is really Jenkins 'killer' feature, since if Jenkins itself doesn't provide what you're looking for (often the case), more often than not there is a plugin that does. Some of my favorites: Cron Column, Rebuild, NodeLabel Parameter, Log Parser, and Email-ext.

  • Advanced scheduling/trigger support: The schedule syntax basically is cron, so you have the same flexbility there, but this is supplemented by Triggers, the REST API, and the Groovy/Java API

Cons

  • Central point of failure: Because all your jobs are kicked off by one server, if that box goes down and nobody notices, Big Trouble. So you better have good monitoring to catch outages immediately, as well as all your configurations saved in source control. Even if you can't get the original server back up, as long as you have your job configs it's trivial to get them setup somewhere else. If time to resolution is a concern, having a standby pre-configured somewhere is probably a good idea as well.

  • If you have multiple environments (Dev, UAT, Prod), typically you have slightly different versions of a job running on each environment. Having all these jobs on one Jenkins can become unwieldly, and manually configuring them gets to be a huge pain. In our case, we run a separate Jenkins 'Cron' instance for each environment. The instances are installed and configured automatically using an in-house deployment tool. You may not have something like that, but there are open source tools which do similar things (generate configs using templates). If you can solve the config generation problem, this makes setting up and deploying Jenkins much easier, and also makes it easier to keep all your stuff in source control.

  • Upgrading Jenkins sometimes breaks functionality, especially with plugins. Don't upgrade your mission critical Jenkins instance until you've tried the new version somewhere else first. This is where having a mirror Dev environment with its own Jenkins instance comes in really handy.

One thing to perhaps emphasize: We do indeed also use Jenkins for CI, but this is a separate instance... the 'cron' instances are dedicated to job management, and the 'CI' instance is dedicated to CI. Separating the concerns seems to make things cleaner.

As a side note, I use Jenkins instead of cron on my Linux box at home :)

By the way, this is actually a pretty common Jenkins use case. For example, Sandia National Lab uses Jenkins this way: https://software.sandia.gov/trac/fast/wiki/Hudson

And there are numerous blog posts and tutorials describing this. Here are a couple examples: http://blog.vuksan.com/2011/08/22/using-jenkins-as-a-cron-server/

http://morgajel.net/2011/12/12/1108

I should also add that this really just pertains to Jenkins, and not all CI tools in general. Just because Jenkins is well-suited to do this does not mean others (TeamCity, buildbot, etc) are...

Improve this answer
8

I would have said that you're not using the right tool for the job here as the main point of CI tools is that they monitor something - your source code typically - and when there's a change they kick off a build/deploy/what ever.

However, these tools can run scheduled jobs (TeamCity does for example), so you can deploy a web site (for example) when there's no one around. So having a single central list of all the tasks you run is in fact a good idea. The tools should also allow you to decide when and how often these jobs are run.

Another benefit is that you can even monitor the system remotely (should you wish).

So, on balance, I'd say it was a sensible thing to do.

Improve this answer
2
  • Your feelings on the subject reflect mine. Because CI is generally known to be for builds and tests, I see it as an unorthodox solution. The other answers to this question definitely have showed that to be the case, as many see it obviously the wrong tool for the job. As TeamCity may perform these additional tasks, any CI tool using Maven projects can do any number of things. I am still uncomfortable that it is a good idea.
    – smp7d
    Mar 7, 2012 at 14:21
  • 1
    @smp7d - agreed. It's a possible solution, but not an ideal solution.
    – ChrisF ♦
    Mar 7, 2012 at 14:23
6

It sounds like cron is already an appropriate tool for your needs. I recommend you start by documenting your system better. Audit the various systems and put together a comprehensive list of which processes run on which machines.

Then consider designating a dedicated machine to run all these cron processes. Make sure you document which machine this is, and assign appropriate admin privileges for controlling it. Put all the the cronjobs on that machine, and then you've got a central point of control for your various automated processes.

Improve this answer
2

My gut reaction is the same, that you are using a tool that has a concept of schedule in it, to do the work of a job scheduler.

You've not mentioned what your jobs are, but your mention of CRON makes me guess they are shell scripts, etc. There are open source and commercial job scheduler packages out there. Sometimes they are referred to as batch schedulers. Some will just wrap up CRON and make it friendlier. Some, like Quartz scheduler, does powerful management of jobs, but requires them to be implemented as Java classes. You could potentially use that, and wrap up the runtime calls to your various scripts using a java wrapper. I believe you'll find plenty of options if you look further.

Improve this answer
1
  • The jobs are a mix of scripts in various languages, java processes, and linux commands. Quartz alone will not give me the front end/build management that Jenkins would provide and I don't want to build all that. I wouldn't be surprised if Jenkins uses Quartz behind the scenes. I will check out this Quartz Manager though (terracotta.org/products/quartz-scheduler).
    – smp7d
    Feb 24, 2012 at 19:18
2
+25

Do not use CI for running periodic tasks that not related to build.

Also avoid cron for tasks that not related to system maintenance.

Use right tools. For application needs - try to use AMQP based solutions.

P.S. I see, that cron fits for your case. On other hand you have a lot of tasks - so try to write supervisor app for them.

Improve this answer
4
  • 1
    Thanks for the answer. Can you describe what you mean by "supervisor app"?
    – smp7d
    Mar 5, 2012 at 14:12
  • In couple of words - it is supervisord.org . Meta program that control status and execution of other processes. You can easily develop your own solution that will fit to your needs. I've a batch of periodic tasks on my project and github.com/ask/django-celery helps me to get out from cron.
    – Nikolay Fominyh
    Mar 5, 2012 at 20:58
  • Thanks, I'll look into Supervisor. The purpose of using the CI tool was to prevent us from needing to write our own tool. The CI tool is slick as can be already.
    – smp7d
    Mar 6, 2012 at 15:52
  • 1
    Guess I don't have the rep to vote this down, but it's a pretty terrible answer--shame it got the bounty. What makes a tool the "right tool"? Even if it has exactly all the needed components, it's the "wrong tool" because it's called a CI system?
    – DougW
    Jun 20, 2014 at 20:40
1

You need to use an enterprise service bus (ESB) for this type of task.

Now my background is in windows/BizTalk, but I am sure that all the equivalents exist on the unix side of things as well. What we would normally do is set up processes on the BizTalk box that would be in charge of kicking off things on the other box, monitoring the progress/errors, and reporting back the status to SharePoint (or web) portal, or sending emails and such if it needs attention.

The benefits of this approach is that all of the configuration and management of your various businesses processes are centrally located, so you know where to start looking. The software already exists that lets you abstract the coding part from the physical config (in BizTalk you can program against a logical 'port' like a sql server, and then in prod, if a sql box changes location or is upgraded or whatever, you can change the configured physical port using their admin tool, again, I am sure equivalents exist on the unix side).

The benefits over using a CI tools would be things like, if your process error out you can automatically, physical resubmit the messages and you can set up a clustered fail-over environment, having a better suited record and logging system; also, once you have the system in place, it would allow you to start architecting your organization to use, or better utilizing SOA. The down side is that depending on the size of your organization the development effort might be high, and licensing costs may be prohibitive.

Improve this answer
3
  • Maybe this is applicable, but I am not sure it is any more a case of applying the wrong tool as CI would be. It is my impression that ESB would be used when communication or process choreography are needed. In this case, we just want central management for an array of standalone processes. We are fine with running custom linux commands though the central management, so any OS/Programming Language agnosticism is probably overkill. This is probably worth looking into though, thanks.
    – smp7d
    Mar 5, 2012 at 15:14
  • If you are a unix shop definitely go with that, I know IBM has a product in their websphere line, and there is webmethods as well that are commercial, and appache has an open source offering; you are right in a sense of your definition of ESB, unfortunately ESB has become somewhat ambiguous in its usage, but consider if you eventually want to add centralized error reporting, or any kind of reporting like 'did it run' into your process that is choreography.
    – aceinthehole
    Mar 5, 2012 at 16:01
  • @smp7d I know that webMethods Integration Server has first class scheduling support. Works well.
    – Rob Grant
    Sep 12, 2014 at 14:47
1

Theoretically it makes sense for you to have a single location for controlling all the disparate jobs, however based on industry experience that is like the "Holy Grail" you will need cron jobs here, bash scripts there and cli scripts over here.

There is also a mantra "If it ain't broke, don't fix it", so while they are plodding along, initially focus on documenting what scripts you have running, what they do and what systems they touch so that you "know" how your business is being run.

Then as a long term strategy setup a centralized system for running the jobs, pick your solution wisely because you are going to have to live with it. Then for each change request, enhancement, upgrade, bug fix or new solution you add within your business architecture ensure that its scheduled and automated tasks are added to your "enterprise control solution".

That way you gradually migrate from a batch of scripts to a more enterprise friendly environment.

Improve this answer
5
  • These are some good thoughts. So you think that what I am looking for doesn't exist and that a CI tool is not a reasonable alternative?
    – smp7d
    Mar 7, 2012 at 13:45
  • It may exist but pragmatism on what you use may cause you to still have cron jobs and bash scripts. However using your CI environment may be an obstacle later because CI is primarily for development workflows, yet as the environment matures you are looking for operations related solutions. Later you may decide to move your version control/CI to the cloud, you do not want that to be bogged down coz its running your organizations day-to-day operations.
    – Stephen Senkomago Musoke
    Mar 7, 2012 at 13:52
  • Well, we were thinking we'd use a separate CI tool for process management, but I see what you are saying.
    – smp7d
    Mar 7, 2012 at 14:06
  • Since you are looking at a separate CI,then why not look at tools focused on process management, monitoring and reporting. That way you leverage the effort for setting up the CI on getting the right tool for the job, if it fails, you have the CI to fall back on
    – Stephen Senkomago Musoke
    Mar 7, 2012 at 14:15
  • I agree that this is the most reasonable path to take. Quartz Scheduler, supervisord.org, and an ESB have been recommended. Do you have any additional recommendations or thoughts on these? (also: When I said separate CI, I just meant another install of our current tool with perhaps some new branding...setup would not be an issue)
    – smp7d
    Mar 7, 2012 at 14:25
0

In large enterprise systems that I worked with, they tend to use a tool designed for scheduling. The most popular one I have used is CA7. It allows you to centralize all the scheduling for all your systems.

Cron is generally used for the single machine, though you can "hack it" by doing ssh remote calls. However, it won't have the concept of dependencies and other stuff. When it comes to operations teams where their scope is even more limited, it's best that a tool is used.

Improve this answer
1
  • Your recommendation led me to this... en.wikipedia.org/wiki/Job_scheduler - Surprisingly nobody ever mentioned this name for such a tool. This may be what I was looking for as if it is designed to do what I am looking for, time will probably show that it does it better than a CI tool. It will take some research to verify that though.
    – smp7d
    Mar 7, 2012 at 20:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.