1

I have a little React app and I'm ready to test it. The first thing I need to do is to create some input objects with random data. I can proceed in one of two ways:

  1. I can create my own fake data line by line using something like faker.js. This will create new fake data every time I run my test. For example:
    let car = {
      Model: faker.lorem.string(),
      Year: faker.number.random(2010, 2020)
    }

The pro of doing this is that I can precisely control all of my input fields. The cons are that it takes longer to write each line of fake data (especially if there are many objects) and that the output is less deterministic.

  1. I can automatically generate all my input up front one time using something like intermock. The pro is that this is much less labour intensive than the first choice. The con is that I lose some control of the data that gets generated.
    // inside car.ts...
    interface Car {
      Model: string
      Year: number
    }

    // in the terminal...
    node ./node_modules/intermock/build/src/index.js --files ./car.ts --interfaces "Car"

Which option is better? Are there any other pros or cons?

1 Answer 1

5

I'll address the general principals rather than these two specific test utilities.

There are different kinds of tests which are for different purposes.

The purpose of a unit test is to quickly indicate if your test subject is working. It tests some example inputs and can include interesting boundary cases.

These tests should be deterministic, otherwise a test failure won't tell you if the test subject is broken vs. the test didn't correctly check a case. You won't like flakey unit tests.

These tests should also run quickly so you can run them every time you change the test subject. A couple seconds or so is quick enough that you won't hesitate to run them early and often.

Unit tests are the single most important type of implementation test. [If you're shipping software to end users, usability tests may well be more important overall.] Because of that, it's more important to quickly create these tests and keep them up to date with test subject changes (e.g. adding fields to the input form) than for these tests to be comprehensive.

The purpose of a regression test is to check that the test subject still works on known good cases, including testing that past bug fixes are still fixed.

These tests also need to be deterministic but they don't need to run as fast as unit tests so they can be more comprehensive.

A deep test [there must be other names for it] can spend a long time testing a wide variety of inputs including combinations of boundary cases and near boundary cases, off-by-one errors, ... and random test cases. For this to be useful, the test program needs to predict the correct output for each test case (otherwise it just checks that the test subject didn't crash, which isn't saying much unless the test subject is in C/C++). That can take a while to implement for random inputs.

A deep test can get more coverage by using nondeterministic inputs. If there are occasional false negatives (test cases identified incorrectly as failures) that's not so bad: You'll need to debug what went wrong anyway, and it could be the test subject or the test output checker.

It's not great to run nondeterministic tests in Continuous Integration since flakey results doesn't tell you whether the code changes are at fault.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.