Do bugs from Fuzzing belong in my system tests?

Question

I have recently implemented fuzz testing over one of my projects. Several bugs have come up, generally classifiable as (dare-I-say trivial?) input validation bugs, and I've fixed them. In no case so far have the breakages involved business logic, which I guess is reasonable.

My question is in the following: When I get a new feature request for the project, I usually write an automated test to describe the desired behavior (using Lettuce, for instance). Do bugs from fuzz testing belong as part of these behavior descriptions?

Myself, I've determined that the advantage of writing explicit test cases is deterministically verifying against regressions. Fuzz testing tends not to be deterministic, though in our case our tests don't have a whole lot of surprising randomness in them.

At the same time, I've written a few of these, and compared with other tests that describe interesting command sequences and results, security bug tests are boring, to the point of creating noise for the reader. Consider the following test shape:

Scenario: Some jerk user doesn't fill in the "home address" form field.
  Given an otherwise correctly-filled form on my web page
  And a "number of hats" field filled with string "banana"
  When the user submits the form
  The application should not crash
  And the user should be redirected back to the form
  And the form should display an error

[repeat for each of 10 fields that could have ridiculous input]

Should I leave this sort of check for the fuzzer which is already catching them through black spells and random opportunity? Or should my regression tests reflect every mistake I've ever made or thought of making?

Answer 1

I don't think there should be too much (read: hardly any) randomness in regression tests. If your buzz test found an error that happens when "bvcyjkfa76!" is entered into "number of hats", make exactly this string (or "banana" if you like it better and it triggers the same error) a test case, but don't let your unit test throw a random string at the program.

The last thing you want is a test that sometimes passes, sometimes fails, just because the random generator sometimes hit the sweet spot and sometimes misses it.

Answer 2

I personally would add the test. If the question is "should I add a test for " I generally will fall on the side of Yes if the test could be informative in some way.

If nothing else by adding the test you know when the bug is fixed.