Anyone know, how many users are valid to use the prototype of the software that we have made? I have conducted research on the reference to the "Software Engineering A Practitioner's Approach book Roger S. Pressman". There is no mention for valid users of software testing?

  • 4
    There is no general answer for this question. It depends entirely upon your software. – Dan Wilson Jan 18 '19 at 19:11
  • 2
    ... and about two dozen other considerations. – Robert Harvey Jan 18 '19 at 20:54
  • This table mentions some types of user tests at the end. As others have said, though, it depends entirely on the nature of the software and the nature of the users. – Nick Alexeev Jan 18 '19 at 22:13
  • 2
    Please read my warning comment below the answer you accepted against applying the results of one study on each and everything. I would also recommend to have a look into 5 worlds of Joel Spolsky and think about in which of those "worlds" you are. – Doc Brown Jan 19 '19 at 9:19

In Usability testing there is a sort of rule, based on an article by Jacob Nielsen, that tells you that 5 users is usually enough to find most issues on a given system. Beyond 5, the new finds are almost negligible and you need to go to 15 users in total to be able to find all problems.

Granted, we are talking about usability and not functional testing, but given that any test on functionality has to follow a set of requirements to know if the system complies or not with the acceptance criteria (in usability you'll have some general heuristics), you can extrapolate the notion that you may not need a lot of users to find most issues within a system.

Of course, if you are testing a gigantic application, using more people could probably help you find bugs faster, but that doesn't necessarily mean that they'll find more.

| improve this answer | |
  • "on a given system" - well, the reference is about web sites exclusively, and it seems to refer to the few systems of the kind and size the authors of that paper have used for their study in 1993. So I recommend against generalizing such statements to all kinds of software, especially against applying it to the OPs system of which we know nothing. – Doc Brown Jan 19 '19 at 9:14
  • No, the studies done by Nielsen (and several others throught the years) are not limited to web sites, nor are they confined to any kind or type of system. Some authors do argue about the number, but between 5 or 10, it is still a small number of users what helps you find most usability issues. As i said, you can use this as a reference, not as a definitive answer as one might not be available. – Juan Carlos Eduardo Romaina Ac Jan 19 '19 at 9:44
  • And size is no issue: for either functional or usability testing, guidelines (or test cases) would be followed. This can limit the scope. – Juan Carlos Eduardo Romaina Ac Jan 19 '19 at 9:45

There is no fixed amount of valid users of a prototype. Because it's not about the number of users but their expectations.

Using prototype-grade software has many potential problems: It

  • will be buggy,
  • might lead to data loss, and
  • will change frequently.

However, having actual users interact with a prototype can be super valuable because that lets the developers see what is really required and whether the software works as expected. This requires some amount of cooperation between users and developers, e.g. the users should be able to send bug reports if they counter a problem.

So the number of users and the openness of a pre-release program needs to balance the potential risks of using prototype-grade software on one hand with the benefit of being able to deliver more useful software more quickly.

A few example scenarios:

A new accounting software is developed. One accounting team of seven people is trained with a prototype software in advance so that they can test it in practice. Risks are mitigated by only transitioning one team as a kind of pilot study. Also, the previous software will still be available if something goes wrong, so bugs won't be business-critical. Developers interview the accounting team members to learn about their experiences and needs.

This kind of approach is great for agile development as directly interacting with individual users lets you discover their real requirements more quickly.

A software as a service web app regularly releases new features. To test new features, they are first deployed to a beta server. Users can opt in to using the beta server. This allows the users to select their acceptable level of risk and acceptable rate of changes. Many organizations need to test new releases before they may be deployed widely. The developers get feedback via analytics, automatic crash reports, and an online support system. Here, the group of beta testers may include many thousands of people.

This kind of approach is useful where you already have concrete features in mind but need to test whether they work in practice, using statistical data. Well known examples of this strategy include the Windows Insider program or the Firefox Beta release channel. Firefox also used to have the Testpilot program where new experimental features could be installed in the browser, without being connected to the browser's release schedule. And here on Stack Exchange, the most recent site redesign was opt-in for a couple of weeks before it became the default.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.