4

Let's say I have a class like this:

    public class Validator  {

        private HashSet<byte> _validFlags;

        public Validator()
        { 
            _validFlags = new HashSet<byte>
            { 
                1, 3, 4, 7, 19, 30, 47//These numbers are chosen for whatever reason and there does 
                                      //not have to be any logic here
            };
        }

        public void Validate(byte inputToValidate)
        {
            if(!_validFlags.Contains(byte))
            {
                throw new ArgumentException("Invalid argument value");
            }
        }
    }

Now, I want to write unit tests for this class.

Positive unit tests are easy for this. However, how about negative unit tests?

Basically, I want to make sure that if I pass anything but specified seven values to this method, I get an ArgumentException. Would passing all the possible byte values other than these seven be an overkill?

I understand I cannot do that if the type of the argument is long or int or float, but what about byte? It would be fairly easy to write a piece of code to populate a structure with all invalid values and just run the test that would expect ArgumentException for all of them. But, again, is that an overkill, both in terms of time needed to write such a test and the time required for that test to be executed (there could be tens or even hundreds of tests like these)? Should I just pick a few invalid values here? I am worried about allowing possible invalid values to pass through the validation because I did not cover them in my negative unit test.

I would appreciate any opinion on this topic.

5
  • Any reason for not using enumerators? They are often better to use than magic numbers not only because they provide compiler support, but because they also offer nice mnemonics for the developer to remember what each value does without needing to hunt for a comment somewhere – T. Sar Jan 18 at 17:05
  • @T. Sar As a general rule, you are correct. However, in this case, these values are sent as numbers from external system. No use in introducing enumerators - the semantics of the numbers is defined in the standard so one should only look it up there. – Vladimir Stokic Jan 18 at 19:26
  • Nothing blocks you from converting numbers to enumerators, tho... – T. Sar Jan 18 at 22:03
  • 1
    @T.Sar: Before a conversion to an enum takes place, I would expect a validation like the shown one to be necessary to validate if the conversion of an external value is allowed. So "converting to an enum" does not solve the problem, it is part of the problem. – Doc Brown Jan 19 at 6:41
  • @T.Sar: From the point of view of code readability, you are absolutely correct and that should be done. However, from the point of negative testing, these values are sent as numbers from an external system and converting them to enums will not solve the problem of testing the validation. – Vladimir Stokic Jan 19 at 7:53
5

I would probably decouple the data part of the validator from the component itself:

    public class Validator  {

        private HashSet<byte> _validFlags;

        public Validator(IEnumerable<byte> validFlags)
        { 
            _validFlags = new HashSet<byte>(validFlags);
        }

        public void Validate(byte inputToValidate)
        {
            if(!_validFlags.Contains(byte))
            {
                throw new ArgumentException("Invalid argument value");
            }
        }
    }

Now this class can be unit tested with all the usual testing techniques you know, some of them mentioned here in the other answers (boundary tests, edge case testing, testing the "happy" paths, the "unhappy" paths, coverage testing, and so on). But it makes clear:

It is not the job of the unit tests for this class to make sure the SUT is validating against the "right" numbers.

The job of the unit tests is to make sure one does not screw anything up in case the internal implementation of the Validator changes, or in case it gets additional requirements (but for this changed Validator version, using "a different set of numbers" is not a different requirement any more).

The right numbers now have to be defined somewhere outside, once, in only one place, maybe in an external text file, to keep the code DRY. Making sure these numbers are picked and encoded correctly is a QA task which refers to pure data, not to code. And "data" which does not follow a specific, algorithmic creation logic, cannot be unit tested sensibly, it needs to be proofread.

Finally, there will be some code in the system which takes the data and hands it over to a new instance the Validator. Still, tests for this piece of code need to check if the handing-over works, not if the data is correct.

TLDR; by making a stronger separation between data and code, you can avoid the need for creating unit tests which are geared towards specific data.

9
  • 1
    I'm not sure I agree that "a different set of numbers" is not a requirement change. A slightly contrived example: if the original spec was "please only allow options 3, 5 and 9, because the others have legal implications", and a change request comes in saying "please also allow option 4 because we've cleared it with our lawyers", I would expect to reflect that change in the test definition. If you can trivially automate a test that covers a large range of "banned" inputs, why not do so? – IMSoP Jan 20 at 11:24
  • 1
    Isn't that just begging the question though? Somewhere, you have some code that defines that list of numbers; that code has requirements; and you want to test that it meets those requirements. So how do you test that code? – IMSoP Jan 20 at 12:54
  • 1
    While all this abstraction sounds very clean, I'm still not convinced it really solves the problem presented in the question, it just moves the pieces around. If anything, it now gives you an even more complex set of tests to write, because you've got to choose both validFlags and inputToValidate in each test case, and come up with edge cases of how they might interact. It's like if you were trying to match valid URLs, so you built a new regex engine; now you have a whole regex engine to test! – IMSoP Jan 20 at 13:53
  • 1
    Sure, it's an exaggerated example, but the point is that generalisation has a cost. In this case, the code is shorter, but the range of inputs is much bigger: you now effectively have input space of 256 ** max_list_len + 1 (byte[], byte) where the original had an input space of 256 (byte). Your tests need to be correspondingly more complex - you now have to test cases like an empty list of allowed values, or one with duplicates, or an implementation that accidentally relies on the allowed list being in order. – IMSoP Jan 20 at 15:25
  • 1
    @IMSoP: yes, that's true. Such design decisions are never black and white. What I suggested avoids magic numbers somewhere buried deeply in a black box, for the price of having a "hole in the box" where the numbers can be injected through. But I am pretty sure that in the long run separating the responsibilities of "validating against a list correctly" and "providing the correct list of numbers" will provide a more maintainable and evolvable solution. – Doc Brown Jan 20 at 15:37
4

You could utilize boundary testing. Test for each positive value. Write one negative test for each value on either side of the valid values.

For example, test that 1 works. Test that 0 and 2 do not. Repeat this strategy until you've tested all valid values.

7
  • Why is it more important to test 0 than e.g. 10? – bdsl Jan 18 at 17:19
  • @bdsl: you test values on either side of each of the valid values. Not just one of the valid values. I was just giving one example. – Greg Burghardt Jan 18 at 17:53
  • 1
    @GregBurghardt this is a valid approach and a necessary one when working with int, long, float... However, here I do not have boundaries. The set of valid values is not a range. It could be something like given in the example: 1. 3, 4, 7, 19, 30, 47. So, I was just wondering about the ideas for approach for negative testing in this case. I know that in some cases you cannot cover all the negative cases, but you have to get close enough. So, how close is close enough? – Vladimir Stokic Jan 19 at 7:48
  • 2
    It's often important to combine boundary cases with representative cases (an arbitrary value somewhere in the middle of the range). I recently came across this Stack Overflow answer where code for selecting singular/plural in translations had been tested for the boundary case of 2=plural, but was actually incorrect for any case above 2. – IMSoP Jan 19 at 11:35
  • 1
    @VladimirStokic: While you have a small list of valid numbers, there is a finite range of values in between. For this reason boundary testing still works, because the mathematics are sound. 1 is valid. 2 is not. 3 is valid. 4 is valid. 5 and 6 are not. 7 is valid. 8-18 are not. You have a series of valid ranges, even if each range only has one number. Again, it is all based on the assumption that if 7 and 19 are valid, then 8 is just as invalid as 9, which is just as invalid as 17 and 18. – Greg Burghardt Jan 19 at 12:19
4

For checking that 249 out of 256 values are invalid, sure, go ahead and write a loop that verifies all of them. It doesn't take that much time on each run of the test suite, and it gives you total peace of mind.

Obviously, if you had a sequence of bytes that had to be exactly right, the tie needed to run the test would quickly become prohibitive and you should sample a subset of the invalid values, perhaps chosen at random to maximise your chances of spotting an error if it should get introduced some day.

2

There a few possibilities

  1. Despite the comment "... numbers are chosen for whatever reason" it's probably more common to have some reason and logic behind them (eg, a Fibonacci or another number sequence), rather than an arbitrary definition. So perhaps you could test that definition logic.
  2. If the numbers arrive from an external system, then maybe it's worth testing the methods passing those numbers to the internal processing logic. (This is somewhat related to this answer)
  3. You may be able, say, with TypeScript, to use the type system, eg by defining a union type 1|3|4|7|19|30|47. Then the compiler does the job, and the tests become unnecessary. As pointed by @DocBrown this or enums suggested by @T.Sar may be inappropriate for numbers coming from an external system. OTOH, if the numbers reflect the domain knowledge encoded within the internal module, then types might be appropriate -- ie, it doesn't matter to the module where the numbers come from.
  4. Generally, property-based testing (PBT) instead of or together with usual tests could help with tricky testing. Good PBT libraries, such as QuickCheck and its ports (see this answer), automate error search for a SUT. They include deterministic tests for common errors, so they always test boundary conditions. They also perform randomised tests, potentially biased towards common errors. PBT could thus be an advantage over combinatorially exhaustive negative tests. Yet, PBT only gives probabilistic evidence, rather than a deterministic proof of correctness. It also incurs a performance penalty versus a few standard tests.

Note that PBT addresses some suggestions made in other answers, such as boundary testing, randomised testing, testing combinatorial code paths; see also this answer.

4
  • 3 should be an answer on its own (with some expansion), because at that point one could reasonably upvote it as the "right" answer – Murph Jan 20 at 15:14
  • @DocBrown does point 3 make more sense now? Please feel free to edit. – schrödingercöder Jan 23 at 13:35
  • @Murph I've expanded the PBT point, which is number 4 now. Does it make more sense? Please feel free to add/edit. – schrödingercöder Jan 23 at 13:37
  • @schrödingercöder yes - I can't remember what the rules now are about multiple answers so I gave you the upvote anyway... – Murph Jan 23 at 18:15
1

My rule of thumb when deciding this is that if the values create a new path of code execution, then it should have their own tests. But if they don't create new code path, then they probably don't need one. Additionally, there may also be boundary conditions (e.g. if you're comparing with inequality operators) that requires special attention which may warrant separate tests around that boundary.

At some point this may generate combinatorially large number of distinct code paths that you can't test all code paths exhaustively, that is fine, but you should select your test cases so it bisects the possible code paths into as many distinct parts to exercise as much variety as possible with special attention to combinations that you felt are most likely to get wrong. At one point, you should feel confident that any further tests are unlikely to give you more confidence in the code, and that's when you should stop.

So given what you have here, you should have one test for values in that list and one test for values that aren't in the list, and that should generally cover your testing need for this class.

If you're combining this set of flags with other sets of flags, that may generate more testing requirements.

3
  • The risk with thinking in terms of code paths is that you end up writing your tests based on your understanding of the necessary code (even if you haven't written it yet), rather than the actual requirements. If a function can take 256 different inputs and has a well-specified result for all of them, I would rather list all of them and let the failing tests tell me where the boundary cases are, than hand-tune the tests and find I'd made the same mistake in my selection of tests as in my implementation of the function. – IMSoP Jan 19 at 11:43
  • @IMSoP: that comment is not completely wrong nor completely right. The risk in wrting tests without using white-box techniques is in testing a lot of things several times which would only require one test, and forgetting other tests which might trigger a different execution path. A good test coverage is achieved by both - black-box and white box testing techniques. – Doc Brown Jan 20 at 6:09
  • @DocBrown I agree, to a point. If each test has a cost - in execution time, or in writing and maintaining a lot of boilerplate - then testing the same thing multiple times would indeed be a problem. But using something like a single test method with a data provider feeding it test cases, it really can be more effort to carefully exclude cases than to just list them all as you think of them. To test a function with 5 boolean inputs and a single output, I would write out a table with all 32 combinations, not spend my time deciding which of those combinations is most "worth" a test. – IMSoP Jan 20 at 9:00
1

It depends upon what level of safety are you expecting? Looping over the whole range of data is (generally) expensive. There are only 2^8 combinations for byte, but this number grows too rapidly. Are you willing to trade time and resources (which aren't free by the way, especially assuming your pipeline runs somewhere on a cloud) for such high a safety?

If no, then you might fallback to randomized testing. Have you ever met Haskell's QuickCheck? Nowadays it is ported to all mainstream languages. It provides less safety instantly, but over time, at a global scale, it approaches the same high level of safety as covering the entire dataset does; statistically, of course. It is also quite smart in generating edge cases.

0

Computers don't get bored.

Asking QA to test all 256 input values by hand would be a bad idea, but it's the kind of automation computers are really good at. Unless there is a significant cost in running the tests (which seems unlikely for a data validator), I see no reason not to test all the valid inputs.

Often, testing tutorials will focus on writing a series of named, hand-crafted test cases; each might be in a separate function, consisting of at least three to four lines of code. I think this is unfortunate, because it makes "adding a test case" feel like a big deal - obviously, doing that for 256 different inputs would be horrible.

Instead, you can use a table-based approach:

  1. Define the test so that the input and expected output are represented as parameters to a generic routine. In this case, the "output" is either throwing an exception or not, so you would have a boolean parameter to represent that.
  2. Write out all the possible inputs, and the output for each. For a validator accepting a byte input, this is simple: the numbers 0 to 255, with true or false against each one.
  3. Call the test routine in a loop with each case in turn, either manually or using your test framework's functionality (e.g. in PHPUnit, you should use a data provider so that each case runs as a separate test).

For real-world examples, this test might only cover one dimension of the input, or depend on some initial state. In that case, it can sit alongside other, hand-crafted, tests for the same function, making sure you still think about possible edge cases based on both the requirements (black box) and the algorithm (white box).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.