I have recently implemented fuzz testing over one of my projects. Several bugs have come up, generally classifiable as (dare-I-say trivial?) input validation bugs, and I've fixed them. In no case so far have the breakages involved business logic, which I guess is reasonable.
My question is in the following: When I get a new feature request for the project, I usually write an automated test to describe the desired behavior (using Lettuce, for instance). Do bugs from fuzz testing belong as part of these behavior descriptions?
Myself, I've determined that the advantage of writing explicit test cases is deterministically verifying against regressions. Fuzz testing tends not to be deterministic, though in our case our tests don't have a whole lot of surprising randomness in them.
At the same time, I've written a few of these, and compared with other tests that describe interesting command sequences and results, security bug tests are boring, to the point of creating noise for the reader. Consider the following test shape:
Scenario: Some jerk user doesn't fill in the "home address" form field. Given an otherwise correctly-filled form on my web page And a "number of hats" field filled with string "banana" When the user submits the form The application should not crash And the user should be redirected back to the form And the form should display an error [repeat for each of 10 fields that could have ridiculous input]
Should I leave this sort of check for the fuzzer which is already catching them through black spells and random opportunity? Or should my regression tests reflect every mistake I've ever made or thought of making?