Software Engineering Stack Exchange is a question and answer site for professionals, academics, and students working within the systems development life cycle. It only takes a minute to sign up.
Sign up to join this community
Has anyone had experience with out-sourced testing? Is it worth the fairly high premium you have to pay?
Can you simply hire some off-shore 'mechanical turks' (who offer the same services, but a lot cheaper) with a testing script as a cheaper alternative to professional software testing services?
Sure, you can hire people who don't know anything about testing to do it but I'd expect it to be about as successful as hiring people who don't know anything about programming to do your development.
Testing services are essentially just consultancies comprising of professional software testers.
Testing is a skill with all the associated knowledge, best practice and so on. Professional systems testers with experience are generally people with the right aptitude for testing who've spent years learning about what they're doing. That simply isn't the case for a random individual.
A good tester will know how to develop test strategies, test plans, and test scripts based on the requirements presented. They know how to accurately assess test coverage and progress, they'll understand how and when regression tests need to be carried out and to what level, they understand why tests need to be repeatable and can log defects in a way that allow developers to reproduce them easily, and they know when there isn't time to test everything (which is pretty much always), how to balance the various pressures in ways which have the best chance of maintaining quality (techniques such as striping).
Ranty Bit
Developers get rightly annoyed when people belittle the skills they have and suggest that it's easily learned but to me this is just the same thing. When programmers ask questions like this showing little thought for what other professionals do or know, what right do they have to expect others to behave differently when it comes to their skills?
Seriously, do a bit of research into what testing really is and what's involved.
Ranty Bit Ends
In terms of the specifics of off-shore testing, it shares many of the same problems as off-shore anything, largely arising from two things: communication and cultural set up.
Communication is an obvious one. I won't go into it much other than to say you need a strategy to deal with it and the main question should be "how will the testers show developers the bugs when the developers are saying they can't see them and wandering over to their desk isn't an option?". Any answer to that question involving e-mail or other text descriptions should be kicked into touch immediately.
Cultural set up is less often talked about but is for me a far more significant factor. By culture I don't mean race, I mean business culture. Most companies I've worked for don't have business processes geared around team members being hundreds of miles away. They don't write specifications in enough detail to allow people to work from them without questioning the person who wrote it (who is now in a different time zone). They operate on the assumption that quick meetings to flesh things out which are missing or unclear are easy to organise.
That's the thing I'd really look at - are your processes really good enough to work with people who can't easily just come and ask a question, and if they're not can you change them so they will work (and do you want to).
The problem with getting someone from outside the company to test your software is that they will not know anything about your software. So you need to make sure you have good documentation that they can test the system with: e.g. requirements specification, test specification, user guides, etc.
They also won't care about your software. So you'd need to have a decent contract in place to ensure the system is tested property.
More importantly you need to have a good rapport with the testers - as you and your developers will be dealing with them a lot.
Ideally they need to be involved as the system is developed. Things change, specs change. They can make impartial and very useful contributions here.
From personal experience outsourcing testing can be a lot more hassle than it's worth. The turn around time between testing, finding and fixing bugs can be huge - days instead of hours. At crunch times in your project - i.e. system test! - you may not want to deal with anybody outside the company, in another time zone and culture.
There are some good answers already on the sort of problems you'll need to handle - so I'm going to focus in on this part: "Can you simply hire some off-shore 'mechanical turks' [...] with a testing script as a cheaper alternative to professional software testing services?" - specifically, the idea of using a testing script as a replacement for experience. I'd strongly recommend that you don't try to hire in an external firm to "just run" test scripts you have written.
It's a common misconception, usually most popular amongst the Standard School, that you can essentially split the "hard bits" out, by getting a test lead to write really detailed test scripts that cheap untrained test monkeys faithfully follow. This works about as well as getting an architect to write really detailed instructions that an army of cheap untrained code monkeys faithfully follow - i.e. not very, but non-technical managers like to see it, because it looks like lots of work is happening.
My experience of watching unskilled testers following scripts is - painful. People really do walk right past the most glaring errors without even seeing them when they're focused on script execution. However, it isn't just a matter of being untrained - detailed procedural test scripts are well known to be an ineffective way of testing. Skilled testers also walk right past defects when they're heavily focused on a test script - the reason many are still effective with that approach is that they don't follow the scripts, they just use them as a rough guide - I know they do, because in many years of working in heavily scripted environments, this is what I've observed. (However, most will deny it if you challenge them directly, because they believe they'll get in trouble with their managers. With justification in some cases.)
In short: scripting won't solve problems with lack of skill, or lack of knowledge of your product.
