-1

When I joined my company as a new comer and I was exploring the unit test suite of the product code. It is using gtest framework. But when I checked all the tests, they were testing the whole functionality by calling real functions and asserting expected output. Below is one such test case as an example:

TEST(nle_26, UriExt1)
{
    int threadid = 1;
    std::shared_ptr<LSEng> e = std::make_shared<aseng:: LSEng >(threadid, "./daemon.conf");
    std::shared_ptr<LSAttrib> attr = e->initDefaultLSAttrib();
    e->setLSAttrib( attr );
    std::shared_ptr<DBOwner> ndb = e->initDatabase(datafile,e->getLogger());
    e->loadASData(ndb);
    e->setVerbose();

    std::shared_ptr<NewMessage> m = std::make_shared<NewMessage>(e->getLogger());
    ASSERT_TRUE(m != nullptr);
    ASSERT_TRUE(e != nullptr);
    m->readFromFile("../../msgs/nle1-26-s1");
    e->scanMsg(m, &scan_callBack_26, NULL);
    std::map<std::string, std::vector<std::string>> Parts = e->verboseInfo.eventParts;
    std::vector<std::string> uris = Parts["prt.uri"];
    ASSERT_EQ(uris.size(), 2 );
    ASSERT_EQ(uris[0] , "mailto:www.us_megalotoliveclaim@hotmail.com");
    ASSERT_EQ(uris[1] , "hotmail.com");
}

I found all the tests in the unit test directory having the same pattern like:

  1. Creating and initialising actual object
  2. Calling actual function
  3. Starting actual daemon
  4. Loading actual database of size around 45MB
  5. Sending actual mail for parsing to daemon by calling actual scanMsg function, etc.

So, all the tests appear more of as functional tests, rather than unit tests.

But, the critical part is, on their official intranet site, they have projected the code coverage percentage of this product as 73%, computed using gcov.

Now, code profiling tools like gcov computes coverage on the following params:

  1. How often each line of code executes
  2. What lines of code are actually executed
  3. How much computing time each section of code uses.

As, these tests are running actual daemon, loading real database and calling actual functions to scan the message, of course, above 3 params will play some role in it, so I doubt it will be completely zero.

But my bothering questions are:

  1. Black box testing also does functional testing just as this, so what's the difference between above and functional test?. In blackbox, testers unaware of the inside code, writes test cases to test the functionalities specific to requirements. How above such kind of tests are different than that? So does gcov generated coverage on this test suite, can be trusted or misleading?

  2. Apparently, gcov code coverage data is based on test suite with all technically incorrect unit tests, does it mean the actual code coverage may be even zero?

  3. In unit test, we mock function calls using google mock-like framework rather than calling actual calls Purpose of unit test is to test the code itself, by smallest unit wise. But above tests, seemingly more like functional tests, can gcov generate reliable code coverage data based on it??

This is haunting me for last two days. So thought to serve on the table for experts.

Awaiting wonderful insights :)

Thanks.

2
  • 5
    gcov does exactly what it does - tells you which lines of code were executed. How you choose to interpret those numbers is up to you. – Philip Kendall Jun 13 at 18:19
  • Agreed, gcov is a tool doing exactly it is supposed to be to. But my question was on gcov coverage number on above kind of ‘unit’ tests which I am witnessing first time. So it’s a new learning for me. – user1228352 Jun 13 at 20:20
4

I agree with @ThomasOwens's anwer as far as you should comply to the companies test strategy.

I disagree with his answer as far as different test types serve different purposes, have different properties and consequences. IMHO it is important to know the differences to be able to implement them for maximum efficiency (with the development process in mind, not some technical performance...). You cannot do this if you do not use the terminology of test types correctly.

What you describe looks to my like a misconception: They call it unit tests because they are implemented using a unit test framework. But as you observed, this are not unit tests, but application tests (or module test at best).

Real unit tests verify the behavior of a unit in isolation. Test that require resources not directly accessible in RAM are not unit test by definition (since they depend on more then the tested code).

It is possible to write unit tests without using mocks, but that would require to implement all business logic in independent units and do the "wiring" through delegation objects that are "to simple to fail". This is hard to do consequently.

Unit Tests should implement the FIRST RTFM rules.
The acronyms resolve to:
https://dzone.com/articles/writing-your-first-unit-tests:

  • Fast
  • Iindependent
  • Repeatable
  • Self validating
  • Timely

and
https://www.youtube.com/watch?v=ybDhy7b2i2o

  • Readable
  • Trustwothy
  • Fast (yes, this is crucial... ;o))
  • Maintainable

None of them applies to the test you describe.

But

That does not mean that the test your company created are bad or useless or anything like that. It just means that they are not unit tests, not more ... not less...

4
  • I do agree with you on most points, matching to my traditional standard specific technical understanding of unit tests. Yes, mocking is not compulsory but most cases, there are function calls made and if they are not mocked, then it shows external dependency and loses the sole purpose of testing strength of that particular unit. So, I think mocking becomes essential here except actual unit itself doesn’t hold any dependency. – user1228352 Jun 13 at 20:58
  • 1
    @user1228352 Maybe you didn't get my point here. I did not argue against the use of mocks... – Timothy Truckle Jun 13 at 21:01
  • But still I have a doubt, can gcov data be considered authentic or trustable if these are not actual unit tests (white box) ? – user1228352 Jun 13 at 21:01
  • @user1228352 "can gcov data be considered authentic or trustable" -- Coverage Data create some trust since they show, how much of the code base is covered with tests. But the more important Question to me is: how much of the requirements is covered by test? Unfortunately there is no metric to measure that. TDD is the only way to keep that high. – Timothy Truckle Jun 13 at 21:07
3

What you are describing seems like the difference between solitary and sociable unit tests. I would have some questions about loading a 45MB database and sending actual mail - the database is quite large and sending email often requires a lot of other pieces. Conceptually, though, it seems like the unit tests are sociable unit tests.

The concerns that I have are mostly around the utility of the tests. It can take a long time to load and reload a 45MB database many times during a test cycle to ensure the proper state before a test. The database and the mail server are things that I'd consider mocking out instead of using sociable tests, in order to improve performance and repeatability of the tests.

The gcov tool appears to be doing exactly what was designed to do, which is measure the lines of code executed by the test. As long as lines of code in tools not developed by the organization aren't showing up, it seems reasonable.

Depending on the context, the distinctions between unit and integration and system tests or between white box and black box tests or other categorizations may not matter at all. In an organization that tests each unit in isolation, perhaps differentiating between solitary and sociable unit tests or treating all unit tests as solitary while integration and system tests as sociable make a difference, but it may not matter to another organization.

The better approach is to look at the overall testing strategy, considering who writes tests, when and how they get executed, and if they are accomplishing their goals of improving product quality and reducing risks. If the tests are achieving their goals and the tools are providing the insights that the team needs to find gaps and close them, the specific terminology doesn't matter so much, especially since there aren't universally accepted definitions.

8
  • Thanks Thomas, have a new insight to me on type of unit tests: solitary and sociable. That was new learning aspect. As, I had only been practising solitary so far. So you mean above kind of test is kind of sociable unit test, not functional test as I thought before. But doesn’t it really defeat the purpose of better design, flow and unit (class/function) level strength. So this gcov number is fine, not misleading, according to you. – user1228352 Jun 13 at 20:26
  • One more thing: I don’t understand why asking questions on stack community brutally downvoted Is considered silly? This has been my frequent experience here. Experts should understand that people are asking questions to learn genuinely, not to be treated as stupid questioner. Learning is continuous process, even experts didn’t born experts, they had to go through querying for help process. – user1228352 Jun 13 at 20:31
  • 1
    @user1228352 The term "unit test" and "functional test" don't have universally accepted definitions. ISO/IEC/IEEE standards do define these terms, but not everyone knows of or follows those standards, so you may not be able to use them for your organization. Focusing on achieving the desired outcomes first and then good test design and implementation second is a better approach than trying to debate over specific terms. Without seeing the specific gcov reports and understanding the software system, I can't say if it's misleading, but based on your description, it could be reasonable. – Thomas Owens Jun 13 at 20:35
  • 1
    @user1228352 As far as voting patterns, everyone can vote as they see fit. Unless there's a comment, your guess is as good as mine. – Thomas Owens Jun 13 at 20:35
  • 1
    "ISO/IEC/IEEE standards do define these terms, but not everyone knows of or follows those standards, so you may not be able to use them for your organization." -- I argue that we, as professionals, should use these terms correctly and correct our colleges if they don't. – Timothy Truckle Jun 13 at 21:15
0

I agree with the other answers already given. But here I'll try to answer the specific questions that you had:

  1. Black box testing also does functional testing just as this, so what's the difference between above and functional test?. In blackbox, testers unaware of the inside code, writes test cases to test the functionalities specific to requirements. How above such kind of tests are different than that? So does gcov generated coverage on this test suite, can be trusted or misleading?

Since a blackbox test does not concern itself with inner workings of a program, and thus should have no knowledge of the source code, code coverage should not apply to it. Of course gcov could generate coverage if you have the source code, but changing the blackbox tests based on it would mean those tests are not really blackbox any more.

  1. Apparently, gcov code coverage data is based on test suite with all technically incorrect unit tests, does it mean the actual code coverage may be even zero?

If you know a test is bad, it should be fixed or removed, so that the final code coverage numbers reflect the amount of code covered by good tests. It doesn't make sense to say you've got 100% coverage if your tests are bad.

  1. In unit test, we mock function calls using google mock-like framework rather than calling actual calls Purpose of unit test is to test the code itself, by smallest unit wise. But above tests, seemingly more like functional tests, can gcov generate reliable code coverage data based on it??

Yes, the code coverage data is reliable. Note that you can have 100% coverage with solitary and/or sociable unit tests, and still miss bugs. If you don't have 100% coverage, then that means you need more tests, but it doesn't tell you what kind of tests you need.

2
  • Another very peculiar thing I found, in unit test suite, these tests are written based on jira id of that stories and test cases specific to that story functionality Should white box testing be independent of functionality and rather code unit? I have an example of such file, let me see if I can attach it here. – user1228352 Jun 13 at 21:10
  • White box testing can be anything I believe. Adding a test for a story about implementing new functionality is fine, even better is if you write the test before implementing the feature; this is test-driven development. Adding a test for a bug ticket is also fine; this is regression testing. – G. Sliepen Jun 13 at 21:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.