71

For example: While performing functional testing of a form in a web application, we will test the fields by entering different kind of random input values.

In general, we as users of the web application do not actually enter random values into fields.

So what is the use of incorporating all those testcases which may/may not lead to bugs, when the probability of appearing these kind of issues in production is way less?

Note: The above example is only a sample case; such issues may occur in any kind of feature/module.

I am asking this question only to know whether any standard practices are there to follow or it totally depends on the product, domain and all other factors.

190

You might not enter random values into fields of a web application, but there certainly people out there that do just that.

Some people enter random by accident and others do it intentionally trying to break the application. In both cases, you don't want the application to crash or exhibit other unwanted behavior.
For the first type of user, you don't want that because it gives them a bad experience and might turn them away.
For the second type of user, they usually don't have honorable intentions and you don't want to let them have access to information that they shouldn't be able to access or allow them to deny genuine users access to your services.

Standard practice for testing is to verify not only that the good-weather case works, but also that unusual edge cases are explored to find potential problems and to have confidence that attackers can't easily gain access to your system. If your application already crashes with random input, you don't want to know what a attacker can do with specially crafted input.

  • 16
    And then there are things that are not people that do it. 👽 – kojiro Dec 16 '17 at 20:44
  • 110
    Or they might be trying to enter their actual legal name, such as “O'Malley”, “姓名” or “Robert'); DROP TABLE Students;-- ”. – l0b0 Dec 17 '17 at 0:27
  • 90
    Or maybe genuine company names, ; DROP TABLE "COMPANIES";-- LTD. – Ben Dec 17 '17 at 16:15
  • 25
    I think the last paragraph can be made stronger by stressing that if a program crashes with random inputs of a cat walking across the keyboard, it will almost certainly crash (and worse) with malicious inputs. – phihag Dec 18 '17 at 1:50
  • 11
    Also, many people enter random input because they don't want to supply the real data (like their name, birthday etc.). Some also assume the computer is as smart as an attendant, and may type something like "last year" instead of "2016" and expect your application to deal with it, just like a human would. – Luaan Dec 18 '17 at 8:59
102

Never Assume Anything

You cannot assume that any user will not do something "dumb" with your software by accident or on-purpose. Users can accidentally press the wrong button, the cat can walk over the keyboard, the system can malfunction, their computer can be hijacked by malicious software, etc.

Furthermore, the user themselves may be malicious, intentionally seeking ways to break your software in the hope that they may find a way to exploit it to their advantage. Even if they find a bug which they can't exploit, anything they find may spur them on to probe your system for something which they can attack, knowing that your QA procedures are lacking.

As far as testing is concerned, it is useful to guard against random inputs, however choosing test inputs entirely at random (i.e. with no particular consideration toward any use case or edge cases) is bordering on useless. The purpose of testing is to validate your solution against the requirements and expectations of your employer/clients/users; this means you need to focus on targeting all edge-cases and boundary conditions, as well as any 'degenerate' cases which don't fit in with a user's expected workflow.

Of course, you might run tests which reveal bugs that you later decide aren't worth fixing; this may be for all kinds of reasons - the bug may be too expensive to fix relative to its impact on the user, or you may discover bugs in features which nobody uses, or the bug may be so well established into the system already that some users are treating it as a feature.

Alternatively, you may be writing some bespoke software which has a tightly limited audience of 'expert' users where there's no commercial benefit to spending time fixing bugs, because those users are capable of doing their jobs with buggy software (for example, a diagnostic tool used by the internal IT team isn't delivering any revenue, so if it crashes occasionally, then nobody is likely to want to pay for the time required to fix it - they'll just tell the IT team to live with the bugs instead).

However, you can only make these decisions if you know about these bugs. For example, a user may enter a malicious input which wipes the entire database - if you haven't explicitly protected against and tested for this scenario, then there's no way you can be sure whether or not this can happen. The risk of leaving undiscovered bugs in the system means that you are potentially leaving yourself open to real problems if one of those bugs reveals itself out in the real world and has a major impact on your users.

So while the decision on whether to fix bugs may require some input from the owner of the software (usually whoever pays your salary), the decision on whether to test for bugs, and which cases to test for is an engineering concern that needs to be factored into the estimates and project planning, where the goal should be for something as close to full coverage as reasonably possible given the constraints on time/money/resources.

  • 12
    Although testing entirely at random isn't useful, and you should certainly explicitly test as many edge-cases as you can think of, a certain amount of random fuzzing can sometimes also be useful to check for issues that you might not have forseen. – Sean Burton Dec 16 '17 at 12:32
  • 10
    We have a saying: "It's so difficult to write idiot-proof software because idiots are such clever people". So, do test for "nonsense" inputs! – Ralf Kleberhoff Dec 16 '17 at 14:36
  • For example, a user may enter a malicious input which wipes the entire database - if you haven't explicitly protected against and tested for this scenario, then there's no way you can be sure whether or not this can happen. Like little Bobby Tables from this XKCD comic? ;) – nick012000 Dec 17 '17 at 10:57
  • 12
    "Never assume anything." I assume this is good advice. – candied_orange Dec 18 '17 at 4:33
  • Thank you for pointing out that not all "bugs" are "fixes". There is a big difference in being aware of an edge case, and spending time and money fixing an edge case. It sure would be great to allow any possible input to a web form and have a set response for all cases, but maybe that isn't relevant to your specific software. Maybe your input only allows numbers on the front end, so it's impossible to receive non-numbers on the back end. "Fixing" the potential bug of having non-numbers in your numbers only form is a waste of time and money. – EvSunWoodard Dec 18 '17 at 19:13
60

There are several factors to take in account. To illustrate those points, I'll use an example of a field where a user should enter a percentage in a context of a quota defined for a specific task in terms of how much disk space the task could use. 0% means the task wouldn't be able to write anything to disk; 100% means the task could fill all the disk space. Values in between mean what they mean.

As a developer, you're probably considering that the acceptable values are [0, 1, 2, 3, ⋯ 99, 100], and everything else is silly. Let's see why users could still be entering those “silly” values.

Typos

%^

The user was entering the value 56, but mistakenly pressed Shift while entering them (for instance because on French keyboard, you have to press Shift to enter digits, and the user was constantly switching between a French keyboard and a QWERTY).

In the same way, you can get a number, with something after or before it, or in between:

56q

Here, the user was probably entering the digits, followed by a tab to move to the next field. Instead of pressing   ⇆  , the user pressed the neighbor key.

Misunderstandings and misinterpretations

An empty input is probably the most usual. The user imagined that the field was optional, or didn't know what to put in this field.

56.5

The user thought that floating point values were acceptable. Either the user is wrong, and the application should politely explain why only integer values are accepted, or the initial requirements were wrong, and it makes sense to let users enter floating point values.

none

The user misunderstood that when asked for the space the task could take, the app expected a number. This could indicate a poor user interface. For instance, asking the user “How much disk space should the task take?” invites to this sort of input, while a field with a percent sign following would receive less of that sort of input, because “none %” doesn't make much sense.

150

The user misunderstood what the percentage means in this case. Maybe the user wanted to tell that the task can take 150% of the currently used space, so if on a disk of 2 TB, 100 GB are used, the task could use 150 GB. Again, a better user interface could help. For instance, instead of having a bare input field with a percent sign appended to it, one could have this:

[____] % of disk space (2 TB)

When the user starts typing, it would change the text on the fly to become this:

[5___] % of disk space (102.4 GB of 2 TB)

Representations

Large numbers or numbers with a floating point can be represented differently. For instance, a number 1234.56 could be written like that: 1,234.56. Depending on the culture, the text representation of the same number would differ. In French, the same number will be written like this: 1 234,56. See, a comma where you wouldn't expect one, and a space.

Always expecting a specific format using a specific locale would get you in trouble sooner or later, because users from different countries would have different habits of writing numbers, dates and time, etc.

Humans vs. computers

Twenty-four

Ordinary humans don't think the same way as computers. “Twenty-four” is an actual number, independently of what a PC would tell you.

Although (1) most systems don't handle at all this type of input and (2) nearly every user wouldn't imagine entering a number written in full letters, it doesn't mean that such input is silly. In About Face 3, Alan Cooper makes a point that not handling such inputs is indicative of the inability of computers to adapt to humans, and ideally, the interface should be able to handle those inputs correctly.

The only thing I have to add to Alan Cooper's book is that in many cases, numbers are written in digits by mistake. The fact that computers expect their users to make mistakes (and won't tolerate a user who writes correctly) is annoying.

Unicode

5𝟨

Unicode reserves its own surprises: characters which could look the same are not the same. Not convinced? Copy-paste "5𝟨" === "56" to the developer tools of your browser, and press Enter.

The reason that those string are not equal is that the Unicode character 𝟨 is not the same as the character 6. This would create a situation where an angry customer would call, telling that your app is not working, providing a screenshot of an input which looks legit, and your app claiming that the input is invalid.

Why would anyone enter a Unicode character which looks like a digit, you would ask? While I wouldn't expect a user entering one unintentionally, a copy-paste from a different source could cause that, and I had a case where the user actually did such copy-paste of a string which contained an Unicode character which wouldn't appear on screen.

Conclusion

Those are the cases you get for an elementary number input field. I would let you imagine what you can have to handle for more complex forms, such as a date, or an address.

My answer is focused on what you called “silly” input. Testing is not about checking the happy paths; it's also about checking that you app doesn't break when a malicious user is intentionally entering weird things, trying to break it. This means that when you're asking for a percentage, you also have to test what happens when the user is responding with a string containing 1,000,000 characters, or a negative number, or a bobby table.

  • 9
    Ah, U+1D7E8: MATHEMATICAL SANS-SERIF DIGIT SIX. – Andreas Rejbrand Dec 16 '17 at 18:37
  • 23
    Re the other unicode character: On Japanese keyboards it is very common to switch from normal digits to full-width digits where a digit is as wide as a kanji. So a Japanese user might have had Japanese input on (rather than English) and accidentally inputted full-width digits. – Jan Dec 16 '17 at 18:51
  • 3
    Before seeing your 5𝟨 section regarding the same homoglyph issue, I was actually expecting a 1 234,56 string (using U+00A0 NO-BREAK SPACE instead of U+0020 SPACE), which is the proper way to codify those number markers (or with U+202F NARROW NO-BREAK SPACE, peroahps). Copying the value from any application that formats the numbers according to the locale before presenting to the user would very easily produce that. – Ángel Dec 16 '17 at 23:04
  • 4
    copy-pasting is a much bigger trouble. Common is to copy paste spaces, line breaks, invisible characters... – Sulthan Dec 17 '17 at 7:54
  • 7
    @Arseni Mourzenko You must be lucky. Copying from e.g. a PDF and pasting is liable to paste all sorts of characters which might be undesirable depending on circs, e.g. ligatures (for fi etc.), smart quotes, en or em dash where ASCII minus was desired, etc. – Rosie F Dec 17 '17 at 19:20
12

There are a lot of good answers here that describe why this is important, but not a lot of advice on how to sensibly protect your application. The "standard practice" is to use robust input validation, on both the client and the server. Non-sensible input is easy to defend against; you simply reject anything that doesn't make sense in that specific context. For example, a social security number consists solely of dashes and digits; you can safely refuse anything else the user types into a social security number field.

There are two kinds of testing that should take place on every application you write, and they each have different purposes. The testing you do on your own application is positive testing; its purpose is to prove that the program works. The testing what testers additionally do on your application is negative testing; its purpose is to prove that your program does not work. Why do you need this? Because you're not the best person to test your own software. After all, you wrote the thing, so obviously it already works, right?

When you write input validation, you will employ positive tests to prove that your validation works. Testers will use random inputs to attempt to prove that it doesn't work. Note that the problem space for random inputs is essentially unbounded; your goal is not to test every possible permutation, but to limit the problem space by rejecting invalid input.

Also note that the end user is not the only one providing input to your program. Every class you write has its own API and its own constraints on what is considered valid input, so robust validation (i.e. "code contracts") is important for your classes also. The idea is to harden your software so that unexpected behavior is rare or nonexistent to the maximum extent possible.

Finally, workflow is important. I've seen applications fall over, not because the user entered something non-sensical, but because they did things in the application in an order that was unexpected. Your application should be aware of this possibility and either handle unexpected workflows gracefully or require the user to perform operations in your specified order.

  • A common example of an application that expects a certain order is a "teardown" function that releases handles that were never reserved. – wizzwizz4 Dec 16 '17 at 19:37
  • 2
    Unfortunately, standard practice is to reject anything that doesn't make sense and leave the user confused and frustrated. Correct practice is to accurately explain (e.g. using error messages/feedback) why the input was rejected, so that the user knows how to correct their input and get it accepted. A simple "get integer from 1 to 100" requires a minimum of 4 different error messages (empty string, unsupported character, too large, too small); plus testing to ensure the right feedback is given in each case. – Brendan Dec 17 '17 at 4:55
  • 2
    @Brendan: Only one message is required: "This must be a number between 1 and 100." User's don't know (and don't need to know) what a string is, or what "unsupported characters" means. Those are programmer affectations, not user help. – Robert Harvey Dec 18 '17 at 17:45
  • @RobertHarvey I'd probably add onto that statement something along the lines of "composed of digits". Because the input "Seventy-Nine" is a number between 1 and 100, but isn't an input most programs can work with. – Delioth Dec 18 '17 at 19:41
  • 1
    @Delioth: You can't fix stupid. – Robert Harvey Dec 18 '17 at 19:53
11

Usually the 'random' values are not random. You are attempting to capture edge cases, the "unknown unknown".

Say for example the # character will crash you app. You don't know this in advance and it would be impossible to write test cases for every possible input. But we can write a test for "¬!"£$%^&*()_+-=[]{};'#:@~,./<>?|\" and see if it breaks

  • 2
    +1 It is surprisingly at first glance how often those random chars will find a bug. Data from user input can do a lot of traveling through many components/service. It only takes one component in the chain not processing it right for the system to have a bug. – Lan Dec 16 '17 at 15:18
  • 4
    esp. now that mobile keyboards all have emoticons – Ewan Dec 16 '17 at 15:25
  • for .Net developers the IntelliTest tool (formerly called Pex) is really good way to exercise code paths to find edge cases, it's especially useful in input validation and for getting good code coverage. – James Snell Dec 17 '17 at 23:14
7

I once wrote a program, which I live-tested in a lab with 60 students. I was standing behind the 60 computer screens and saw them use it. The amount of ridiculous things they did was hair-raising. I was drenched in sweat watching their "creativity". They did much more than any single individual can fantasize up within a lifetime. Of course one of them broke it.

After that I follow an approach: if "a very specific use case" do, else show error

If I have several use cases, I strictly define them and chain the above.

  • 1
    However, those specific use cases might very well be too specific. We always underestimate the space of valid inputs. (O'Hara, locally formatted decimals etc). How many financial routines were prepared to handle negative interest rates? – Guran Dec 19 '17 at 13:22
6

What you're describing is Fuzzing or Fuzz Testing: throw random and invalid input at a system and see what happens. You don't do this because you expect a user to do it. You do it to expose your own assumptions and biases to stress the edges of your system to see what happens.

Normal test input written by a human with will come with assumptions and biases. These biases can be certain classes of bugs are never found via testing.

For example, if most of your input is in the ASCII-safe Unicode range, assumptions about the character encoding in the code aren't exercised. Or maybe it's always below a certain size, so a fixed sized field or buffer isn't hit. Or maybe there are special characters which get interpreted in surprising ways exposing that user input is being fed to a shell or used to build queries in an unsafe manner. Or maybe there's just too much "happy path" testing and not enough attempts to exercise the error handling.

Fuzzing has no such preconceptions about input. It will brutally exercise your system with any possible combination of "valid" input. Unicode, ASCII, big, small, and lots and lots of errors. Your system should respond gracefully to all of them. It should never crash. The user should always get some sort of sensible message about what went wrong and how to fix it. It's not Garbage In / Garbage Out, it's Garbage In / Error Out.

While one might dismiss the resulting explosions because "no real user will do that", that misses the point of the exercise. Fuzzing is a cheap way to eliminate your biases about possible inputs. It's a cheap way to throw all the weird things users will try to do at your system. As the engineer, your job is ensure your system fails gracefully.


Futhermore, fuzzing "input" isn't just about users. It could represent be the result of an API query to a 3rd party service, what if that starts sending messed up results? How does your system deal with that? A proper system should alert an admin that a component has gone bad. An improper system will quietly reject the bad query, or worse, accept it as good data.

Finally, some users are malicious. If you don't fuzz test your system, someone else will. They will probe the edges of your system for common mistakes and try to use them as security holes. Fuzz testing can simulate this, to an extent, and you can deal with any possible security holes discovered before they become a problem.

  • And there's the Quick Check testing tools which do similar things – icc97 Dec 19 '17 at 8:43
4

If your aim is to create a quality product then test every possible type of input that a user will be physically able to submit. Otherwise you're just waiting for the day when someone submits that one type of input you didn't feel needed testing.

During a big demonstration of new e-auction software at a local authority where I worked, my manager decided (admittedly with some mischief) that he felt the need to see what happened if he put in an auction bid with a negative value. To my genuine surprise the auction software allowed this nonsensical bid and the entire auction process ground to a halt. The type of auction being demonstrated should never have allowed negative amounts to be submitted.

Some of the large group of assembled procurement and finance officers were annoyed with my manager for submitting a nonsensical value. But others, including myself, were annoyed with the software developers for failing to test and reject such an obvious type of invalid input. I can only imagine how weak the software must have been at deflecting other types of invalid input (code injection attempts, exotic characters not representable in the database table, etc).

Were it up to me, I'd have returned the software and deemed it unfit for purpose. The difference between a weak and a strong software product is the level of testing it has been subjected to.

  • 2
    test every possible type of input that a user will be physically able to submit. -- That problem space is essentially infinite, and you're wasting your time by trying to test it all. Checking for negative inputs is a single bifurcation; it's not only sensible but also expected from competent developers. You don't have to check every negative number to prove that such validation works. – Robert Harvey Dec 16 '17 at 15:48
  • 13
    That's why I said every type of input, and not every possible input. And I'll repeat my point: if you don't test every type of input, the users eventually will. – Arkanon Dec 17 '17 at 16:32
1

For example: While performing functional testing of a form in a web application, we will test the fields by entering different kind of random input values.

Yes. This is a kind of test but it's not a functional test. This is what is called stress testing. It is the act of applying pressure to a system to see if it can handle it.

So what is the use of incorporating all those testcases which may/may not lead to bugs, when the probability of appearing these kind of issues in production is way less?

When you are stress testing software you're trying to discover the boundaries of what the software's limits are.

The tests are in a way exhaustive by nature. Where you need to discover usage limits, breaking points, check all logical branches or see how partial failures affect the entire system.

You can have all your functional tests pass, but still fail stress testing.

I am asking this question only to know whether any standard practices are there to follow or it totally depends on the product, domain and all other factors.

Yes, this is a standard practice.

Testing software is about asking a question about expected behavior, and when all the tests pass this communicates that the software operates as intended. This is why tests make for good pre-conditions on deploying updates.

Stress testing doesn't provide clear specific pass or fail indicators. The results are more informative. It tells you what your system can handle and you make decisions from that information.

You can define specific goals for stress testing which must be passed in order to continue to the next stage in development. Those can be included as part of your quality assurance process, but changes in the environment can change the results of a stress test. So people run stress tests at different times to see how the system handles changing conditions.

What I mean is you can't just run stress testing everytime you deploy a new version of your software, and assume this means everything will pass stress testing later.

protected by gnat Dec 17 '17 at 9:26

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.