I am on C++ and using gtest as the main framework. Say I have a edge detection function I want to test that takes an image as an input and returns the edge detected image. I have 3 images ready to be checked.

Is it better to write it in a way that is self contained like

void EdgeCheck(const std::string& input, const std::string& ans) {
    cv::Mat in_img = cv::imread(input);
    cv::Mat ans_img = cv::imread(ans);
    // Do some checks here
}

TEST(Edge) {
    EdgeCheck(std::string("path to image 1"),std::string("path to ans image 1"));
    EdgeCheck(std::string("path to image 2"),std::string("path to ans image 2"));
    EdgeCheck(std::string("path to image 3"),std::string("path to ans image 3"));
}

int main(int argc, char** argv) {
    return UnitTest::RunAllTests();
}

so the test can be executed by simple ./unit-test or is it better to write it like

TEST(Edge) {
    cv::Mat in_img = cv::imread(argv[1]);
    cv::Mat ans_img = cv::imread(argv[2]);

    // Do some check here
}

int main(int argc, char** argv) {
    return UnitTest::RunAllTests();
}

but it has to be executed by

./unit-test image1 ans_image1
./unit-test image2 ans_image2
./unit-test image3 ans_image3

From my understanding, the advantage of the first approach according to UnitTest++ command line arguments is "a well-written unit-test is self-contained, and parameterizing a test means the test is no longer just a test. It's now a function rather than a unit test."

However, the advantage of the second approach is it's more flexible in the sense that if I want to add image4 and image5, I don't need to recompile the code and it can be done on a bash script, but then it makes the test not self contained.

Which approach is considered a better practice?

  • 1
    How do you specifically define "better" in this context? – Robert Harvey Jul 11 '17 at 21:45
  • 1
    Robert Harvey is correct. You have to decide if it is better or not based on your own feeling. The command line based approach will make it easier to add or remove images from the testing library -- you could iterate all present in a directory. – Frank Hileman Jul 11 '17 at 23:22
  • I wouldn't view this as a question of good vs. bad practice. It's mostly just a question of what sort of test this is. As @Dymeng (somewhat obliquely) pointed out, you seem to be dealing with functionality of a somewhat larger scale than most people think of for a unit test. Testing a subsystem (or entire system) is good practice too; it's just not a unit test. – Jerry Coffin Jul 12 '17 at 0:27
  • @JerryCoffin Testing a simple edge detection function (Ex: Sobel) in a class is too simple to be called an integration test. If using an image as an example is too confusing, you can treat the input as a few integers. Do you think these integers should be hard coded in the test or be entered via the command line? – user3667089 Jul 12 '17 at 2:32
  • 2
    That's a bit abstract, isn't it? How about this: which is more important to you; a more flexible approach, or a self-contained test? – Robert Harvey Jul 12 '17 at 2:49
up vote 2 down vote accepted

This is all about how you want to manage your tests and integrate them into your whole environment. I assume you have the requirements to

  • add new tests easily
  • run all your tests at once, maybe as part of an test suite
  • make sure one failing test does not prevent the execution of other, independent tests
  • get a log afterwards which of your tests haved failed, and which not
  • run a subset of your tests or a single test to verify it does not fail any more after you applied a bugfix
  • make it easy for anyone else to run your tests without special knowledge

Looking at both of your two solutions I think making them parametrized by command line will give you more flexibility for switching between running them individually vs. "all at once", and it will make it easier to make sure a failing test does not influence other tests. Maybe it will also be easier to integrate them into a bigger test suite, if you have that requirement. However, this flexibility does not come for free. Since one needs to know the parameters, your command line program, when called in a wrong fashion, should give a descriptive message which kind of parameters it expects, and the test execution script should be integral part of your tests.

So as long as you treat the command line program together with the script as a self-contained component, and as long as you provide some integral documention, there is nothing wrong with the parametrized approach. If that is not "self-contained enough" for you for some weired reason, integrate things in one program, but live with the fact the other requirements I mentioned may need more effort to resolve.

Automated tests can be parameterized without depending on passed in parameters. Often a hybrid approach is most useful.

For example if run without any command line args, your test might scan a configured directory iterate over all the images in that directory.

It may be useful in this case to be able to override the directory in this case, and providing a command line parameter or specifying an environmental variable are common ways to do that. Or users can simply place additional images in the default directory to extend your tests.

A well-written unit-test is self-contained, and parameterizing a test means the test is no longer just a test. It's now a function rather than a unit test

@Dymeng is correct that you're test is probably actually an integration test not a unit test, but that's really beside the point. Unit tests can be parameterized and so can integration tests.

The point about being self-contained is important. You should be able to run tests on your project without specifying command line args because if others who want to run your tests, it would require special knowledge to know the correct parameters to run the test with. But that fact shouldn't deter you from exposing ways to run your test on other data sets.

  • Testing a simple edge detection function (Ex: Sobel) in a class is too simple to be called an integration test. But I like your idea about it should be self-contained while being able to add tests easily. Let me think about this more on how to design this. – user3667089 Jul 12 '17 at 2:38

Sorry to say, but that is the worst idea I heard today, for several reasons:

  • anyone who executes unit tests, has to pass correct parameter at correct position, otherwise the unit tests are going to fail
  • gtests are taking their own optional parameters, and because of that, you can not be sure where the tests parameters are (argv[1] or other index)

The best for unit tests if to actually put data in test (in a variable), and (if possible) not load from a file.

According to The Art of Unit Testing by Roy Osherove, he states the following properties of unit tests.

1.2. Properties of a good unit test

A unit test should have the following properties:

  1. It should be automated and repeatable.
  2. It should be easy to implement.
  3. Once it’s written, it should remain for future use.
  4. Anyone should be able to run it.
  5. It should run at the push of a button.
  6. It should run quickly.

By having command line arguments, your tests are no longer repeatable or automated. Can I go to your computer, press "Run tests" and have them run? The answer is simply no.

I would suggest finding another way to test using those images. Perhaps have them be part of your project, and hard code the relative path?

  • If I have a bash script that I can run in a single click that has all the command line arguments in it, is it better? – user3667089 Jul 13 '17 at 20:46
  • If you are Unit Testing Function1(param a, param b) for example, and you want to test it with multiple inputs, that could be one way. Im not noo familiar with the unit testing framework you are trying to use. I usually do something like this for C# stackoverflow.com/questions/4818484/… – Nicolas Jul 14 '17 at 14:45

I think for unit testing, you shouldn't have any images at all. What the quote you noted is trying to say is that if you're inputting an external reference for unit testing, you're not really unit testing anymore.

I'm not exactly sure what you mean by edge detection, but I'm guessing there's some piece of work within your code that analyzes the image properties (else why have an image?) then uses the results of that analyization to perform some logic against.

A unit test should be given a set of mock results from the image analyzation to test how the logic is performed against it.

An Integration Test, on the other hand, would accept images as a more broad type of test. In which case, you're much better off passing the image data in as parameters (otherwise you're just hardcoding dependencies in your integration tests, which is no good).

So, I think it's not a question so much of how to pass the values for the image paths, but rather understanding that there's a fundamental difference between unit testing and integration testing, and what you're looking at is integration testing (which is why the quote doesn't match up for you: it's talking about unit tests).

  • 1
    While correct, this answer doesn't have much to do with the question. – Frank Hileman Jul 11 '17 at 23:23
  • Testing a simple edge detection function (Ex: Sobel) in a class is too simple to be called an integration test. If using an image as an example is too confusing, you can treat the input as a few integers. Do you think these integers should be hard coded in the test or be entered via the command line? – user3667089 Jul 12 '17 at 2:30
  • Not sure why the downvotes really... it answers his question in that if you're using params like that, you're not unit testing but integration testing, and even notes that he's better off passing the params in rather than hardcoding them (for integration testing). Anyway... – jleach Jul 12 '17 at 15:33

Your Answer

 
discard

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.