I have recently implemented fuzz testing over one of my projects. Several bugs have come up, generally classifiable as (dare-I-say trivial?) input validation bugs, and I've fixed them. In no case so far have the breakages involved business logic, which I guess is reasonable.

My question is in the following: When I get a new feature request for the project, I usually write an automated test to describe the desired behavior (using Lettuce, for instance). Do bugs from fuzz testing belong as part of these behavior descriptions?

Myself, I've determined that the advantage of writing explicit test cases is deterministically verifying against regressions. Fuzz testing tends not to be deterministic, though in our case our tests don't have a whole lot of surprising randomness in them.

At the same time, I've written a few of these, and compared with other tests that describe interesting command sequences and results, security bug tests are boring, to the point of creating noise for the reader. Consider the following test shape:

Scenario: Some jerk user doesn't fill in the "home address" form field.
  Given an otherwise correctly-filled form on my web page
  And a "number of hats" field filled with string "banana"
  When the user submits the form
  The application should not crash
  And the user should be redirected back to the form
  And the form should display an error

[repeat for each of 10 fields that could have ridiculous input]

Should I leave this sort of check for the fuzzer which is already catching them through black spells and random opportunity? Or should my regression tests reflect every mistake I've ever made or thought of making?

  • P.S. Tried to add the "fuzz-testing" tag, but I'm too new, here. :) Feb 15 '11 at 9:46

I don't think there should be too much (read: hardly any) randomness in regression tests. If your buzz test found an error that happens when "bvcyjkfa76!" is entered into "number of hats", make exactly this string (or "banana" if you like it better and it triggers the same error) a test case, but don't let your unit test throw a random string at the program.

The last thing you want is a test that sometimes passes, sometimes fails, just because the random generator sometimes hit the sweet spot and sometimes misses it.

  • What? I never suggested that the regression test would act randomly. The question is whether dozens of ridiculous/trivial security scenarios belong in regression tests, or whether to leave it for later fuzzing. Feb 15 '11 at 12:52

I personally would add the test. If the question is "should I add a test for " I generally will fall on the side of Yes if the test could be informative in some way.

If nothing else by adding the test you know when the bug is fixed.

  • True! That's actually how I verify the fixes. Feb 15 '11 at 12:53
  • And more to the point, you know that if the bug comes back in the future you're certain to catch it. Feb 15 '11 at 13:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.