27

I've been involved with many projects in several companies because I've been a developer for a long time and I'm a contractor.

I estimate that less than 20% of projects are methodically tested. With methodically tested I mean any testing beyond ad-hoc no plan testing.

I also estimate that less than 10% of projects are thoroughly methodically tested where they have dedicated testers as part of the team, test plan document, where developers write automated tests and then they also track test coverage and measure results.

Two questions

  1. What are your percentage estimates about this issue?
  2. What's your professional experience regarding software testing?

Additional note

Since methodical testing question may get quite biased answers (people like to brag about being superior to others) I encourage other developers (those that're not exposed to methodical testing) to provide their answer as well, because otherwise it will look like testing is being done everywhere... except at your company.

Improve this question
0

15 Answers 15

Reset to default
11

The pattern I have seen with testing over my career shows a strong correspondence with the risk of failure in a project. Big projects are more likely to be tested than small ones, mission critical applications are more likely to be tested than one off marketing web sites, in house systems are less likely to be tested than public facing ones.

That said there are still projects that have been excessively tested and those that have not been tested enough, but these are the minority.

Improve this answer
3
  • I've edited my question slightly. Could you provide your estimates as well?
    – Robert Koritnik
    Sep 15, 2010 at 12:44
  • Almost all the projects (80%+) I have been involved with have been methodically tested, but then I have almost exclusivly done corporate mission critical applications.
    – Martin Brown
    Jan 10, 2011 at 12:32
  • I work for pharmaceutical corporation. I would say that 80% of applications are tested by professional testers and developers. Those 20% are low risk applications like promotional presentation on iPad. but even this one is ad-hoc checked by someone.
    – yoosiba
    Feb 20, 2011 at 19:43
7

Everything we produce gets completely tested. If our internal QA team is overloaded, we have an offshore team that tests the projects. They're not as good as our internal team but that's a different topic.

Improve this answer
3
  • I've edited my question slightly. Could you provide your estimates of the development professional market. Because your projects get scrutinised. And are your tests automated or not?
    – Robert Koritnik
    Sep 15, 2010 at 12:46
  • Who is this offshore team? Would you give them a recomendation?
    – Martin Brown
    Jan 10, 2011 at 12:33
  • Usually big companies have R&D Centers in other countries(Mostly in Asia). Where the offshore development is done. The objective of offshore development is to reduce development cost (some part of it).
    – Nipuna
    Apr 27, 2011 at 14:18
5

Yes.

The amount of testing is in proportion to the reliability required of the app, as well as the maturity of the programmer culture.

Web sites are quite often walking bug-holes (broken links are a defect).

Video games are often buggy.

Windows (finally) is fairly reliable.

Routers are very reliable

Hospital monitors "don't break"

Note that the fiscal cost of failure is also correlated to reliability.

Improve this answer
4
  • 2
    I strongly disagree with you - you've never seen a router fail? Do Xbox, Playstation and Wii games lock-up? Ever had a blue-screen or 'Application is not responding' in Windows?
    – JBRWilkinson
    Jan 21, 2012 at 9:33
  • 2
    @JBRWilkinson I think your missing his severity modifiers as well as possibly ingoring the vast majority of PC games which are, as Paul says, often buggy. Anyway, the list could probably use some improvement, but the sentiment is correct: reliability often correlates to the fiscal losses associated with failure.
    – Jae Carr
    Dec 1, 2014 at 18:36
  • @JBRWilkinson: All your examples are for non-predictable issues, often for devices for which "just reboot it" is already the going advice. Testing tends to focus on this that always go wrong, i.e. behavioral issues; or commonly encountered issues. The issues you point out are commonly experienced by users in general, but any given user doesn't experience it remarkably often.
    – Flater
    Mar 17, 2020 at 13:37
  • My comment was 8 years ago; the stability of software has massively improved in that time through practices like Automated Testing, CI, CD, Chaos Engineering, Fuzz Testing, etc. There are also new threats which were entirely predictable but required 'thinking outside of the box' - e.g. could anybody have predicted that IoT devices with poor security would be compromised and become part of the largest botnet in the world? - I think they could have. (csoonline.com/article/3258748/…)
    – JBRWilkinson
    Mar 24, 2020 at 14:07
4

In 10 years, I never worked on a project with formal code testing.

In my current job, we only have functional testing.

The problem is that no one in management is even aware of code testing. The testing department doesn't even know about testing code -- at a high level, they just follow the high level specifications and verify that we comply with them, from a behavioral/functional standpoint.

We don't have a qualified software leader who forces us to code well. The result is spaghetti code, lots of regressions, missed schedules and so on...

Improve this answer
3
  • Thak you for your honest answer. What are your estimates (check my edited question)?
    – Robert Koritnik
    Sep 15, 2010 at 12:47
  • My estimate for Italy is under 10% of formal code testing. Perhaps almost only mission-critical code.
    – Wizard79
    Sep 15, 2010 at 12:56
  • I've worked in Ireland, England, Scotland and Slovenia and as it seems Italy doesn't seem any different.
    – Robert Koritnik
    Sep 15, 2010 at 17:54
3

The three companies I have worked for during the last 15 years all had unit tests which were run automatically.

At two of those companies I pushed for introducing them.

Improve this answer
3
  • I edited my question slightly. Could you provide your estimates about software testing on professional market as well?
    – Robert Koritnik
    Sep 15, 2010 at 12:47
  • @Robert: I haven't been job-hopping enough to give a general estimate. From what little I know, things are becoming better. But then, it had been me who was pushing for automatic unit tests in two out of the three cases I told you about...
    – sbi
    Sep 15, 2010 at 18:23
  • But you do talk to other developers in other companies, don't you?
    – Robert Koritnik
    Sep 15, 2010 at 19:00
3

In the last 9 years, i've basically only met acceptance/regression tests. There were only a few unit tests.

Improve this answer
1
  • I've edited my question slightly. Could you provide your estimates of software testing in professional market as well?
    – Robert Koritnik
    Sep 15, 2010 at 12:48
3

My sample is very small to deduce percentages from, but here goes anyway.

One was a fabless chip + firmware company, which did fanatical testing. 24/7 automated tests on tens of installations, each testing tens of units in parallel. Software teams dedicated to developing testing software. Hardware teams dedicated to building test rigs. Compatibility testing against tens of competitors. Heck, they even bought a multi-million dollar chip tester installation to develop and debug some of the tests that the fabs run when the chips leave the foundry.

Another one was a bank. This one is a completely different environment: no product releases, but lots and lots of in-house software to keep running continuously. These guys tested the cr*p out of every single change they made. They had very strict separation of DEV/QA/PROD environments, automated regression testing, mandatory QA testing signed off by end-users before releasing into production, etc.

So yeah, people do do methodical testing. But as you can tell I've never worked at a place that ships your typical GUI software for the typical computer user.

Improve this answer
3

I currently write embedded firmware for a small startup company making wireless medical devices. We are required to do rigorous testing, and have a completely separate quality department headed by someone who reports directly to the CEO. I have never had my code so thoroughly tested before by separate testers (the only time that compares, is when I was working on satellite TV systems about 15 years ago.)

Our test results get submitted to the FDA (so far we have gotten two FDA clearances -- each submittal was around 500 pages long). Both our development and testing methodologies are both subject to periodic auditing.

So it is not only the big companies that do lots of formal testing.

Note -- in my 25+ years of contract programming/consulting, I have also worked for many companies that did virtually no formal testing. Most of them are not around anymore.

Improve this answer
2
  • I'm also at a medical device company, and the introduction to GMP (Good Manufacturing Practices, FDA-speak for a controlled design/test process) was quite eye-openIng for me. It's made me a better engineer (and docbook expert unfortunately)
    – Bill Gribble
    Jan 21, 2012 at 20:09
  • Everyone who has released code into a regulated environment is familiar with the concept of a quality system. Your quality system might be might be "we show up and fake it the best we can" or you might have one that is written down, but every project has a quality system, implicit or explicit. When it's important to your customers or your business, you pony up and do the professional thing and write down the quality system (even if it's on the back of a cocktail napkin) and follow it, producing "object evidence" of what you did.
    – andy mango
    Mar 12, 2020 at 2:41
2

We are a mid sized offshore company in South Asia. However, we always do USA based projects and directly work with requirements sent from USA company.

We apply methodical testing on every application we build. Perhaps, the quality of testing isn't up to standard, but we do employ them.

Improve this answer
2
  • Would those tests be automated or mainly manual? I've edited my question slightly. Could you provide your estimates of software testing in professional market as well?
    – Robert Koritnik
    Sep 15, 2010 at 12:49
  • Most of our testing is done manually.
    – Shamim Hafiz - MSFT
    Sep 30, 2010 at 10:31
2

As much as the purist in me doesn't want to accept that there has to be some risk management built into the decision for how rigorously you test or whether you do formalized testing at all. For internal apps, which I suspect are a large % of programming projects, the cost of releasing a bug then quickly patching it after it is noticed can sometimes be outweighed by the cost of a full testing team. Of course it depends on the app and the potential cost of failures.

That said, I don't think risk management planning is the reason for the lack of formalized testing. I think it is more a result of non-technical managers not understanding the value it provides and only see the cost.

Improve this answer
4
  • 3
    I hear what you are saying, but it's hard to justify. Studies have shown that the longer a bug goes unnoticed the more it costs, exponentially. The cost of bugs making it to the customer is enormous, and patching them often causes new bugs, if a unit testing framework is not in place (a likely scenario when this kind of "patch and fix" mentality is present). Consequently, judicious use of testing tools and methodologies can be shown to be far less expensive than patch and fix.
    – Robert Harvey
    Sep 15, 2010 at 17:38
  • 3
    I'm growing more and more skeptical of that dogma, especially how it is universally applied. I'm sure it is true some of the time, but not all bugs are created equally, nor are all apps. I find it very hard to swallow that a bug in an internal app used by 10 people is significantly more expensive to fix if it is found during unit testing versus in a patch release. It may be exponentially more embarrassing, but not more costly unless you ignore the real costs of the time a tester spends finding the bug.
    – JohnFx
    Sep 15, 2010 at 18:00
  • 2
    I also wonder if those statistics weren't more applicable to the massive projects (e.g. Operating Systems) that they were created from and don't translate to the CRUD type apps most of us build for a living.
    – JohnFx
    Sep 15, 2010 at 18:02
  • 1
    I agree with both of you guys and have seen both cases. But I have certainly seem my share of the exponential costs Robert is describing, particularly when a bug has been in the software so long that other features would actually start to break if it was fixed. With shoddy enough frantic coding with people working around issues long enough and bugs that stay inside a long enough time, 1+1 is not 2. It's 7. And if it's not 7, everything will fall apart.
    – user204677
    Jan 31, 2018 at 19:14
0

Almost every company I have been did methodical testing. My current company has some basic unit style tests and that is not sufficient. We have had some quality issues due to this. I highly recommend independent thorough testing on any project that is going to be used by anyone besides yourself. The money spent will be well worth it. Applications that don't work don't get used. That goes for internally facing as well as externally facing.

Improve this answer
2
  • I've edited my question slightly. Could you provide your estimates of software testing in professional market as well?
    – Robert Koritnik
    Sep 15, 2010 at 12:49
  • @Robert: I don't understand your question "estimates of software testing". Are you asking my opinion on how many companies test? My estimate would be maybe 90% or more based on what I've seen with my own eyes. Testing is a common part of professional development.
    – Bryan Oakley
    Jan 20, 2012 at 18:49
0

In the last twenty or so years of my career through eight or so companies, I've never worked on a project that didn't do testing. The amount of testing differed at each company, but every professional development project I've ever worked on did formal testing. This applies equally to both small and middle sized companies (where "small" means less than 10 employees, and "middle sized" means a couple thousand employees or less).

Some companies didn't have much automated testing, some didn't have much manual testing, but they had at least one or the other.

Improve this answer
0

It depends on the needs of the customer. In a contract situation there are probably acceptance tests. In-house is usually a slapjob with little testing. Consumer stuff is usually highly covered in frequent functionality but rough around the edges.

Improve this answer
0

Short answer: Yes

Long answer:

  1. I do not have a good estimate for the first category (it is probably some distance from zero, but how much?), but my experience actually corroborates with your second estimate. It is hard to give meaningful percentages since the amount and type of testing depends on the kind of application being developed, and available timeframe and as well as the skillset of the developers and how the project is run. In practice the most important hurdle for the developers would be the acceptance test since that is an important milestone for billing purposes. But it is also the time where the unexpected may happen (more requirements) and the developers may be pressured to deliver and get by with whatever adhoc testing that is possible and timely (at this stage), on top of time needed for troubleshooting and overcoming the unexpected.

  2. I have been through a variety of projects with different combination of factors mentioned above:

    • no formal unit tests, only integration tests and mostly adhoc tests

    • very formal ranging from unit tests to detailed test plans involving dedicated QA resources, automated testing (as conducted by the testers with their own set of tools) and code coverage reports. But these are not always meaningful to the developers as much as to the managers

On the individual level I seek to maintain an understanding of my options when it comes to writing proper tests appropriate for the technology I am dealing with, and exercise them at my own discretion. Basically the things that are actually meaningful and beneficial to my work, and not so much cranking out numbers.

Improve this answer
0

Consider, for example, the CPAN library which is the core body of contributed software for the Perl programming language ...

Any time and every time you install a package from this [vast ...] library, an automated suite of tests provided by the package author is run on your machine. If any of those tests fail, the package will not be installed.

And then, there is "CPANTS." https://cpants.cpanauthors.org/

CPANTS is a testing service for CPAN distributions. One of its goals is to provide some sort of quality measure called Kwalitee. Though it looks and sounds like quality, higher Kwalitee score doesn't always mean a distribution is more useful for you. All it can assure is it's less likely for you to encounter problems on installation, the format of manuals, licensing, or maybe portability, as most of the CPANTS metrics are based on the past toolchain/QA issues you may or may not remember.

Yep ... this automated service constantly tests new CPAN package releases, on all sorts of widely-different hardware and software platforms (that the package is supposed to support ...) and tests if it actually does so.

Of course, this isn't the only library (or, language) that routinely does this.

Very long ago, I realized the personal value of "test-driven development." Write a class, then write a test for it. As you continue to add to the system, constantly run all the tests.

"Suh-prize!!"

Wow ... how did we break this? ... well, I sure am glad we caught that one early! It would have been very nasty had it made it to production!!

Uh huh. Once you learn that lesson for yourself – even when you're the only developer on the project – you don't forget it.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.