Can we assume while testing software that a user wouldn't perform such silly actions on software?

Question

For example: While performing functional testing of a form in a web application, we will test the fields by entering different kind of random input values.

In general, we as users of the web application do not actually enter random values into fields.

So what is the use of incorporating all those testcases which may/may not lead to bugs, when the probability of appearing these kind of issues in production is way less?

Note: The above example is only a sample case; such issues may occur in any kind of feature/module.

I am asking this question only to know whether any standard practices are there to follow or it totally depends on the product, domain and all other factors.

Answer 1

You might not enter random values into fields of a web application, but there certainly people out there that do just that.

Some people enter random by accident and others do it intentionally trying to break the application. In both cases, you don't want the application to crash or exhibit other unwanted behavior.
For the first type of user, you don't want that because it gives them a bad experience and might turn them away.
For the second type of user, they usually don't have honorable intentions and you don't want to let them have access to information that they shouldn't be able to access or allow them to deny genuine users access to your services.

Standard practice for testing is to verify not only that the good-weather case works, but also that unusual edge cases are explored to find potential problems and to have confidence that attackers can't easily gain access to your system. If your application already crashes with random input, you don't want to know what a attacker can do with specially crafted input.

Answer 2

Never Assume Anything

You cannot assume that any user will not do something "dumb" with your software by accident or on-purpose. Users can accidentally press the wrong button, the cat can walk over the keyboard, the system can malfunction, their computer can be hijacked by malicious software, etc.

Furthermore, the user themselves may be malicious, intentionally seeking ways to break your software in the hope that they may find a way to exploit it to their advantage. Even if they find a bug which they can't exploit, anything they find may spur them on to probe your system for something which they can attack, knowing that your QA procedures are lacking.

As far as testing is concerned, it is useful to guard against random inputs, however choosing test inputs entirely at random (i.e. with no particular consideration toward any use case or edge cases) is bordering on useless. The purpose of testing is to validate your solution against the requirements and expectations of your employer/clients/users; this means you need to focus on targeting all edge-cases and boundary conditions, as well as any 'degenerate' cases which don't fit in with a user's expected workflow.

Of course, you might run tests which reveal bugs that you later decide aren't worth fixing; this may be for all kinds of reasons - the bug may be too expensive to fix relative to its impact on the user, or you may discover bugs in features which nobody uses, or the bug may be so well established into the system already that some users are treating it as a feature.

Alternatively, you may be writing some bespoke software which has a tightly limited audience of 'expert' users where there's no commercial benefit to spending time fixing bugs, because those users are capable of doing their jobs with buggy software (for example, a diagnostic tool used by the internal IT team isn't delivering any revenue, so if it crashes occasionally, then nobody is likely to want to pay for the time required to fix it - they'll just tell the IT team to live with the bugs instead).

However, you can only make these decisions if you know about these bugs. For example, a user may enter a malicious input which wipes the entire database - if you haven't explicitly protected against and tested for this scenario, then there's no way you can be sure whether or not this can happen. The risk of leaving undiscovered bugs in the system means that you are potentially leaving yourself open to real problems if one of those bugs reveals itself out in the real world and has a major impact on your users.

So while the decision on whether to fix bugs may require some input from the owner of the software (usually whoever pays your salary), the decision on whether to test for bugs, and which cases to test for is an engineering concern that needs to be factored into the estimates and project planning, where the goal should be for something as close to full coverage as reasonably possible given the constraints on time/money/resources.

Answer 3

There are several factors to take in account. To illustrate those points, I'll use an example of a field where a user should enter a percentage in a context of a quota defined for a specific task in terms of how much disk space the task could use. 0% means the task wouldn't be able to write anything to disk; 100% means the task could fill all the disk space. Values in between mean what they mean.

As a developer, you're probably considering that the acceptable values are [0, 1, 2, 3, ⋯ 99, 100], and everything else is silly. Let's see why users could still be entering those “silly” values.

Typos

%^

The user was entering the value 56, but mistakenly pressed Shift while entering them (for instance because on French keyboard, you have to press Shift to enter digits, and the user was constantly switching between a French keyboard and a QWERTY).

In the same way, you can get a number, with something after or before it, or in between:

56q

Here, the user was probably entering the digits, followed by a tab to move to the next field. Instead of pressing   ⇆  , the user pressed the neighbor key.

Misunderstandings and misinterpretations

An empty input is probably the most usual. The user imagined that the field was optional, or didn't know what to put in this field.

56.5

The user thought that floating point values were acceptable. Either the user is wrong, and the application should politely explain why only integer values are accepted, or the initial requirements were wrong, and it makes sense to let users enter floating point values.

none

The user misunderstood that when asked for the space the task could take, the app expected a number. This could indicate a poor user interface. For instance, asking the user “How much disk space should the task take?” invites to this sort of input, while a field with a percent sign following would receive less of that sort of input, because “none %” doesn't make much sense.

150

The user misunderstood what the percentage means in this case. Maybe the user wanted to tell that the task can take 150% of the currently used space, so if on a disk of 2 TB, 100 GB are used, the task could use 150 GB. Again, a better user interface could help. For instance, instead of having a bare input field with a percent sign appended to it, one could have this:

[____] % of disk space (2 TB)

When the user starts typing, it would change the text on the fly to become this:

[5___] % of disk space (102.4 GB of 2 TB)

Representations

Large numbers or numbers with a floating point can be represented differently. For instance, a number 1234.56 could be written like that: 1,234.56. Depending on the culture, the text representation of the same number would differ. In French, the same number will be written like this: 1 234,56. See, a comma where you wouldn't expect one, and a space.

Always expecting a specific format using a specific locale would get you in trouble sooner or later, because users from different countries would have different habits of writing numbers, dates and time, etc.

Humans vs. computers

Twenty-four

Ordinary humans don't think the same way as computers. “Twenty-four” is an actual number, independently of what a PC would tell you.

Although (1) most systems don't handle at all this type of input and (2) nearly every user wouldn't imagine entering a number written in full letters, it doesn't mean that such input is silly. In About Face 3, Alan Cooper makes a point that not handling such inputs is indicative of the inability of computers to adapt to humans, and ideally, the interface should be able to handle those inputs correctly.

The only thing I have to add to Alan Cooper's book is that in many cases, numbers are written in digits by mistake. The fact that computers expect their users to make mistakes (and won't tolerate a user who writes correctly) is annoying.

Unicode

5𝟨

Unicode reserves its own surprises: characters which could look the same are not the same. Not convinced? Copy-paste "5𝟨" === "56" to the developer tools of your browser, and press Enter.

The reason that those string are not equal is that the Unicode character 𝟨 is not the same as the character 6. This would create a situation where an angry customer would call, telling that your app is not working, providing a screenshot of an input which looks legit, and your app claiming that the input is invalid.

Why would anyone enter a Unicode character which looks like a digit, you would ask? While I wouldn't expect a user entering one unintentionally, a copy-paste from a different source could cause that, and I had a case where the user actually did such copy-paste of a string which contained an Unicode character which wouldn't appear on screen.

Conclusion

Those are the cases you get for an elementary number input field. I would let you imagine what you can have to handle for more complex forms, such as a date, or an address.

My answer is focused on what you called “silly” input. Testing is not about checking the happy paths; it's also about checking that you app doesn't break when a malicious user is intentionally entering weird things, trying to break it. This means that when you're asking for a percentage, you also have to test what happens when the user is responding with a string containing 1,000,000 characters, or a negative number, or a bobby table.

Answer 4

There are a lot of good answers here that describe why this is important, but not a lot of advice on how to sensibly protect your application. The "standard practice" is to use robust input validation, on both the client and the server. Non-sensible input is easy to defend against; you simply reject anything that doesn't make sense in that specific context. For example, a social security number consists solely of dashes and digits; you can safely refuse anything else the user types into a social security number field.

There are two kinds of testing that should take place on every application you write, and they each have different purposes. The testing you do on your own application is positive testing; its purpose is to prove that the program works. The testing what testers additionally do on your application is negative testing; its purpose is to prove that your program does not work. Why do you need this? Because you're not the best person to test your own software. After all, you wrote the thing, so obviously it already works, right?

When you write input validation, you will employ positive tests to prove that your validation works. Testers will use random inputs to attempt to prove that it doesn't work. Note that the problem space for random inputs is essentially unbounded; your goal is not to test every possible permutation, but to limit the problem space by rejecting invalid input.

Also note that the end user is not the only one providing input to your program. Every class you write has its own API and its own constraints on what is considered valid input, so robust validation (i.e. "code contracts") is important for your classes also. The idea is to harden your software so that unexpected behavior is rare or nonexistent to the maximum extent possible.

Finally, workflow is important. I've seen applications fall over, not because the user entered something non-sensical, but because they did things in the application in an order that was unexpected. Your application should be aware of this possibility and either handle unexpected workflows gracefully or require the user to perform operations in your specified order.

Answer 5

Usually the 'random' values are not random. You are attempting to capture edge cases, the "unknown unknown".

Say for example the # character will crash you app. You don't know this in advance and it would be impossible to write test cases for every possible input. But we can write a test for "¬!"£$%^&*()_+-=[]{};'#:@~,./<>?|\" and see if it breaks

Answer 6

I once wrote a program, which I live-tested in a lab with 60 students. I was standing behind the 60 computer screens and saw them use it. The amount of ridiculous things they did was hair-raising. I was drenched in sweat watching their "creativity". They did much more than any single individual can fantasize up within a lifetime. Of course one of them broke it.

After that I follow an approach: if "a very specific use case" do, else show error

If I have several use cases, I strictly define them and chain the above.

Answer 7

What you're describing is Fuzzing or Fuzz Testing: throw random and invalid input at a system and see what happens. You don't do this because you expect a user to do it. You do it to expose your own assumptions and biases to stress the edges of your system to see what happens.

Normal test input written by a human with will come with assumptions and biases. These biases can be certain classes of bugs are never found via testing.

For example, if most of your input is in the ASCII-safe Unicode range, assumptions about the character encoding in the code aren't exercised. Or maybe it's always below a certain size, so a fixed sized field or buffer isn't hit. Or maybe there are special characters which get interpreted in surprising ways exposing that user input is being fed to a shell or used to build queries in an unsafe manner. Or maybe there's just too much "happy path" testing and not enough attempts to exercise the error handling.

Fuzzing has no such preconceptions about input. It will brutally exercise your system with any possible combination of "valid" input. Unicode, ASCII, big, small, and lots and lots of errors. Your system should respond gracefully to all of them. It should never crash. The user should always get some sort of sensible message about what went wrong and how to fix it. It's not Garbage In / Garbage Out, it's Garbage In / Error Out.

While one might dismiss the resulting explosions because "no real user will do that", that misses the point of the exercise. Fuzzing is a cheap way to eliminate your biases about possible inputs. It's a cheap way to throw all the weird things users will try to do at your system. As the engineer, your job is ensure your system fails gracefully.

---

Futhermore, fuzzing "input" isn't just about users. It could represent be the result of an API query to a 3rd party service, what if that starts sending messed up results? How does your system deal with that? A proper system should alert an admin that a component has gone bad. An improper system will quietly reject the bad query, or worse, accept it as good data.

Finally, some users are malicious. If you don't fuzz test your system, someone else will. They will probe the edges of your system for common mistakes and try to use them as security holes. Fuzz testing can simulate this, to an extent, and you can deal with any possible security holes discovered before they become a problem.

Answer 8

If your aim is to create a quality product then test every possible type of input that a user will be physically able to submit. Otherwise you're just waiting for the day when someone submits that one type of input you didn't feel needed testing.

During a big demonstration of new e-auction software at a local authority where I worked, my manager decided (admittedly with some mischief) that he felt the need to see what happened if he put in an auction bid with a negative value. To my genuine surprise the auction software allowed this nonsensical bid and the entire auction process ground to a halt. The type of auction being demonstrated should never have allowed negative amounts to be submitted.

Some of the large group of assembled procurement and finance officers were annoyed with my manager for submitting a nonsensical value. But others, including myself, were annoyed with the software developers for failing to test and reject such an obvious type of invalid input. I can only imagine how weak the software must have been at deflecting other types of invalid input (code injection attempts, exotic characters not representable in the database table, etc).

Were it up to me, I'd have returned the software and deemed it unfit for purpose. The difference between a weak and a strong software product is the level of testing it has been subjected to.

Answer 9

For example: While performing functional testing of a form in a web application, we will test the fields by entering different kind of random input values.

Yes. This is a kind of test but it's not a functional test. This is what is called stress testing. It is the act of applying pressure to a system to see if it can handle it.

So what is the use of incorporating all those testcases which may/may not lead to bugs, when the probability of appearing these kind of issues in production is way less?

When you are stress testing software you're trying to discover the boundaries of what the software's limits are.

The tests are in a way exhaustive by nature. Where you need to discover usage limits, breaking points, check all logical branches or see how partial failures affect the entire system.

You can have all your functional tests pass, but still fail stress testing.

I am asking this question only to know whether any standard practices are there to follow or it totally depends on the product, domain and all other factors.

Yes, this is a standard practice.

Testing software is about asking a question about expected behavior, and when all the tests pass this communicates that the software operates as intended. This is why tests make for good pre-conditions on deploying updates.

Stress testing doesn't provide clear specific pass or fail indicators. The results are more informative. It tells you what your system can handle and you make decisions from that information.

You can define specific goals for stress testing which must be passed in order to continue to the next stage in development. Those can be included as part of your quality assurance process, but changes in the environment can change the results of a stress test. So people run stress tests at different times to see how the system handles changing conditions.

What I mean is you can't just run stress testing everytime you deploy a new version of your software, and assume this means everything will pass stress testing later.