We have a high load SAAS product and due to lack of resources the board has outsouced reporting and OLAP & Warehouse work to a 3rd Party Company. When we started working with them, they accidentally DOS'ed our production system, now they have made too many long running queries on the production databases and slowed the system down to unuseable. They are about to deliver and we have a maintainence contract with them. How do you deal with 3rd companies you can't really trust no to bring your production system down? And what do you do about it when they do? Has anyone any experience of this? (note they have a test environment set up to do everything they need to on).

I'm not looking for answers on how to prevent someone bringing down our system, I'm after what to do with the 3rd party company itself.

  • 14
    Ummm. Don't give them access to your production system maybe? Give them access to a mirrored copy of the data if they need it. – JohnFx Sep 29 '11 at 14:20
  • @JohnFX this isn't my decision, I have no control over this and I know how to prevent it. – Mr Shoubs Sep 29 '11 at 14:29
  • 8
    I'm not looking for answers on how to prevent this, I'm just after anyone else's experiences -- That condition makes this question "Not Constructive." See programmers.stackexchange.com/faq#dontask – Robert Harvey Sep 29 '11 at 14:37
  • 3
    @Mr Shoubs -Your question contradicts itself, the title says you want a way to deal with it, the last line says you just want us to hug you and tell you it will all be okay. – JohnFx Sep 29 '11 at 14:41
  • 2
    Your edit makes the question better, and I voted to reopen, but I still think it's rhetorical, and you already have the answer: the cure is prevention. If a third-party company brings down your servers, and it is not a case of abuse, it's your fault. If it's abuse, you pursue the usual remedies of breach of contract, etc. – Robert Harvey Sep 29 '11 at 15:05

3rd party vendors have to be held to the same standards that in-house would be. That should include qa testing, it should include performance specs that they have to meet. And it should include consequesnces for failure.

The best solution is to document the problems and ask management to fire them. That's what we did when we outsourced something and they broke production with their code fix when it was deployed (for a different client than the one they were working on even try explaining that to your clients). The board needs to know about their incompetence and fire them just as they would any other type of contractor. If they hired an airconditioning firm to come fix the broken airconditioner and they flooded the building by accident and then the airconditioning worked even worse when they were done, would they keep using them too?

  • These are my thoughts, however this company has already done most of the work and taking over now would take up a lot of time. Clients are waiting... did you ever get any kind of compensation, or is it tough and you just don't use them again? – Mr Shoubs Sep 29 '11 at 14:54
  • 2
    I think it depends on the level of bad and how much the company is willing to go to the mat to get it. Usually that involves expensive legal action. – HLGEM Sep 29 '11 at 15:27
  • 2
    Your answer is good. I would add something about documenting the costs to the business of the problems caused by the 3rd party. This cost can now become part of a cost-benefit analysis of outsourcing this work via doing it in house. – Shane Wealti Sep 29 '11 at 19:24
  • All of the above are good ideas ... if the contract you have with the sub-contractor covers non-performance. Without such a contract they are just a wish list and a learning lesson for writing up the next contract. – mattnz Sep 30 '11 at 6:09

Commercial reality Lessons, Subcontracting 101:

Guideline 1) Subcontractors are usually professionals who want you to succeed and will do their best to make it happen.

Rule 1) Subcontractors are in it to make money. They care as much about your business, software and systems to do enough to receive the payment of their next invoice.

Rule 2) Not all contractors are created equal.

Rule 3) If you believe any different you are setting yourself up for disappointment and failure.

Make it worth the subcontractors while not to bring it down, or conversely, not worth there while to do so. Contractually penalty clauses are useful to remind them that they have to perform to certain standards, as is withholding payment unless / until certain criteria are met.

Technically there is absolutely nothing you can do. If you give them access to your production system, they can break it at any time, in any imaginable and unimaginable way.


You need to measure them against a known standard and when they fail to meet that standard then you should have the option to renegotiate or terminate. Here's a link from garter that might provide some usable insight, Measuring the Performance of Outsourcing Engagements

If these decisions were dictated by management based upon a price per hour measure then your only recourse will be to illustrate to them in specific, objective, measurable, verifiable terms how much this decision is costing over and above the price per hour of the solution. Where you have very ineffective outsourced resources you can actually have a negative return on investment, even if the price per hour for the resource was a dollar per hour: Outsourcers cost X; our delays, downtime and opportunity costs related to the site have cost Y. Where Y>X, get rid of the outsourcer.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.