I'm leading a largish development team (~35 developers). We are doing primarily Web Development work on a number of sites.

Historically the knowledge on the teams has been pretty siloed. If you worked on Site A you will know how to troubleshoot it, but you would not be a lot of help on Site B.

We also have a few cross-cutting concerns, i.e. common components used between sites which require specialized knowledge to troubleshoot.

With all this in mind, I'm trying to understand the best way to setup an on-call team.

This would be a team of programmers who would be available to deal with out-of-hours emergency issues occasionally (say one call every 2 weeks). They may be required to deploy emergency fixes.

Part of me is saying we can't have a big on-call team with shallow knowledge, instead we need a smaller team with deep knowledge who can expect to be on-call more often and remunerated as such.

Does anyone have any suggestions based on experience on how to setup this team?

Thanks in advance.

  • 3
    The need for out-of-hours emergency patches is usually a sign of extreme organizational problems. Care to expand a bit on why you need an on call team anyway?
    – yannis
    Nov 2, 2012 at 11:05
  • What YannisRizos says is right. This seems like more of a problem for system administrators not software developers. If software developers need to write emergency code patches in the middle of the night and apply them to production immediately then you have some pretty serious problems.
    – maple_shaft
    Nov 2, 2012 at 11:07
  • 1
    @YannisRizos, Companies that deploy frequently and have software that is used 24/7 internationally often have these needs. For these types of companies, I would claim this is good organizational planning and not "extreme organizational problems".
    – cdkMoose
    Nov 2, 2012 at 16:31
  • 1
    I can't speak for OP's environment, but my team deploys new functionality on as frequent as bi-weekly basis. We can't expect sysadmins to be up to date on all of our changes (and every other team like us in the company). We don't get called often and we seldom actually patch production code immediately, but when that is needed, it is critical and I don't want just anyone dealing with a production issue at 3 AM. It is well worth identifying the right people in the first place.
    – cdkMoose
    Nov 2, 2012 at 16:33
  • These devs may be handling level 2 calls that don't always require programming to solve or caused the system to crash.
    – JeffO
    Nov 2, 2012 at 16:45

1 Answer 1


Where I work, the management first asked for volunteers, and then divided the on-call times based on the amount of available people. In practice, there's three on-call periods for each person during a two month period. They run in a weekly schedule of:

Person 1: Monday 08:00 until Friday 08:00 (64 hours of on-call)
Person 2: Friday 16:00 until Monday 08:00 (64 hours of on-call)

During office hours (08:00 - 16:00) there is no on-call, of course, as calls will go directly to the on-duty phone.

During cases where the on-call person is unable to solve the issue, he can call the technical lead and the overtime hours will start running for the Technical Lead, and the on-call person continues his on-call, though usually working together with the Technical Lead until the issue is fixed.

Update: Oh and the on-call person only handles issues related to his area of the system. If the issue goes beyond that, there is a different on-call person in another team. For example the Unix team has a separate on-call person, and the Database team has one, etc.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.