Is this testing practice unit testing, regression testing, and/or functional testing?

Question

In a project, its program is written in C++, and implements some http service.

The testing program for the project is written in Python. The Python testing program doesn't call the C++ project program at source level, but loads and executes the executable compiled from the C++ project program, and interacts with the project's executable by http request and response.

My question is about what type of testing the python testing program is.

It looks to me that the python testing program tests at executable level not at source level, so it is black-box testing, which is also called functional testing if I am correct.

But I was surprised to hear that someone said it was unit testing and regression testing. I thought it was wrong.

1. About unit testing, is it correct that,

   • unit test is to test a function or procedure,

   • similarly to black-box testing, unit test doesn't care about the implementation of the function or procedure,

   • but unit testing is to test the function or procedure at source code level but not at executable level,

   • so unit testing program must be written in the same programming language as the program to be tested?

2. About regression testing,

   the project's C++ program originally had its Python testing program.

   Then I started to add new features to the C++ program, and created my own Python testing program to test the new features only.

   I heard that the original Python testing program was still run automatically by Jenkins, but I haven't checked it. Does the combination of my new testing program and the original testing program count as regression testing?

Thanks.

Answer 1

Don't get too hung up about terminology. These terms are not precisely defined, and they are not necessarily exclusive.

Unit testing is definitely the wrong term, since it refers to testing smaller parts of the code in isolation. The "unit" does not necessarily refer to a function or procedure though, it could also be a class or even multiple classes. (Unit tests does does not have to be written in the same language, as long as the language is is able to directly call/access the code in the unit under test. But usually it will be.)

But either functional test, system test, integration test, end-to-end-test and regression test could be appropriate terms for what you are doing.

Integration testing refers to testing integration between units or subsystems, while end-to-end testing refers to testing the entire system. But these terms are most appropriate when testing a larger system which consists of multiple subsystems. It sounds like this is not really the case for your program, so I would probably just say system test.

Regression testing is a test specifically written with the purpose to verifying that existing functionality keep working correctly after changes are introduced. This is different that say writing tests along with development to verify new and changed functionality. So unit tests or integration tests or functional tests can also be regression tests. Basically it is just a question of when you write the test - if you write the test after you already know the functionality works correctly, then it is a regression test!

The term white box and block box testing refers to how much knowledge of implementation details you have when writing the test. If you wrote the program yourself, you probably have some ideas about what is the trickiest parts of the code and this may give you ideas about what conditions to test. This is white-box testing, even if you are only testing the system through the public interfaces. If you wrote the code yourself, you cannot write black-box tests.

Functional tests refers specifically to testing to verify the program conforms to the requirements specification. How much of your test is functional test depends on how much is written in this document! Requirements are typically written at a high level, describing what the customer should be able to do with the system rather than how the system should be implemented, so functional tests will be black-box tests.

Answer 2

This is basically integration testing. It's definitely not unit testing as you are using an additional piece to test it (the python).

Whenever you test the final product of your application - in this case a C++ executable - you are moving into either integration/functional tests.

loads and executes the executable compiled from the C++ project program, and interacts with the project's executable by http request and response.

This is not at all unit testing. Unit testing is testing against the units within a piece of software, not against their compiled result.

Regression testing doesn't make sense either, unless all of the tests in python were written due to bugs and the point is to prevent those from regressing. Though this term is heavily overloaded and depending on what your coworker means by it, it might be applicable (or not). If your definition of "regression testing" is "testing software to find bugs" then any testing can count. If it's more specific? Then it's less so.

Honestly, overall this sounds like you are only running API integration tests from python. That's not necessarily the best idea, you probably want to do some level of unit testing within the C++ codebase itself.

Answer 3

The person who told you this is unit testing is completely wrong.

Unit tests are done, as you call it, at "source code level" (although I'd say that's a pretty vague description - after all, tests run code and only compiled code can be run, unless it's interpreted).

Unit tests rely heavily on mocking, that is, faking out components (dependencies) of a component (e.g. a single class) that you want to test.

What you are talking about in your question is something I call end-to-end testing. It's not an integration test either because those are still done without using an external program. Both integration and unit tests are usually performed using a specific test framework and test runner for the platform at hand.

Other terms that may apply for this case: system testing, acceptance testing (depending on what metrics are being validated).