In a project, its program is written in C++, and implements some http service.

The testing program for the project is written in Python. The Python testing program doesn't call the C++ project program at source level, but loads and executes the executable compiled from the C++ project program, and interacts with the project's executable by http request and response.

My question is about what type of testing the python testing program is.

It looks to me that the python testing program tests at executable level not at source level, so it is black-box testing, which is also called functional testing if I am correct.

But I was surprised to hear that someone said it was unit testing and regression testing. I thought it was wrong.

  1. About unit testing, is it correct that,

    • unit test is to test a function or procedure,

    • similarly to black-box testing, unit test doesn't care about the implementation of the function or procedure,

    • but unit testing is to test the function or procedure at source code level but not at executable level,

    • so unit testing program must be written in the same programming language as the program to be tested?

  2. About regression testing,

    the project's C++ program originally had its Python testing program.

    Then I started to add new features to the C++ program, and created my own Python testing program to test the new features only.

    I heard that the original Python testing program was still run automatically by Jenkins, but I haven't checked it. Does the combination of my new testing program and the original testing program count as regression testing?


  • 1
    "Regression testing" is just using tests to ensure that new functionality hasn't broken existing code. You can do this with tests at multiple abstraction levels. Testing the complete system from outside I would refer to as "end-to-end", but the terminology isn't very precisely defined. – jonrsharpe Oct 30 '16 at 14:47

Don't get too hung up about terminology. These terms are not precisely defined, and they are not necessarily exclusive.

Unit testing is definitely the wrong term, since it refers to testing smaller parts of the code in isolation. The "unit" does not necessarily refer to a function or procedure though, it could also be a class or even multiple classes. (Unit tests does does not have to be written in the same language, as long as the language is is able to directly call/access the code in the unit under test. But usually it will be.)

But either functional test, system test, integration test, end-to-end-test and regression test could be appropriate terms for what you are doing.

Integration testing refers to testing integration between units or subsystems, while end-to-end testing refers to testing the entire system. But these terms are most appropriate when testing a larger system which consists of multiple subsystems. It sounds like this is not really the case for your program, so I would probably just say system test.

Regression testing is a test specifically written with the purpose to verifying that existing functionality keep working correctly after changes are introduced. This is different that say writing tests along with development to verify new and changed functionality. So unit tests or integration tests or functional tests can also be regression tests. Basically it is just a question of when you write the test - if you write the test after you already know the functionality works correctly, then it is a regression test!

The term white box and block box testing refers to how much knowledge of implementation details you have when writing the test. If you wrote the program yourself, you probably have some ideas about what is the trickiest parts of the code and this may give you ideas about what conditions to test. This is white-box testing, even if you are only testing the system through the public interfaces. If you wrote the code yourself, you cannot write black-box tests.

Functional tests refers specifically to testing to verify the program conforms to the requirements specification. How much of your test is functional test depends on how much is written in this document! Requirements are typically written at a high level, describing what the customer should be able to do with the system rather than how the system should be implemented, so functional tests will be black-box tests.

| improve this answer | |
  • I don't get how's most competent answer is downvoted like this. – Aleksei Zabrodskii Oct 30 '16 at 19:01

This is basically integration testing. It's definitely not unit testing as you are using an additional piece to test it (the python).

Whenever you test the final product of your application - in this case a C++ executable - you are moving into either integration/functional tests.

loads and executes the executable compiled from the C++ project program, and interacts with the project's executable by http request and response.

This is not at all unit testing. Unit testing is testing against the units within a piece of software, not against their compiled result.

Regression testing doesn't make sense either, unless all of the tests in python were written due to bugs and the point is to prevent those from regressing. Though this term is heavily overloaded and depending on what your coworker means by it, it might be applicable (or not). If your definition of "regression testing" is "testing software to find bugs" then any testing can count. If it's more specific? Then it's less so.

Honestly, overall this sounds like you are only running API integration tests from python. That's not necessarily the best idea, you probably want to do some level of unit testing within the C++ codebase itself.

| improve this answer | |
  • Thanks. About unit testing, is unit testing to test the function or procedure at source code level but not at executable level? Must unit testing program be written in the same programming language as the program to be tested? – Tim Oct 30 '16 at 15:19
  • Is regression testing only tied to bug fixing tests? – Laiv Oct 30 '16 at 15:19
  • @Tim you can't be testing at the unit level if you are by definition testing a compiled, executable level. – enderland Oct 30 '16 at 15:21
  • @Laiv I clarified that section. – enderland Oct 30 '16 at 15:22
  • does functional testing or blackbox-testing perform at executable level or source code level? – Tim Oct 30 '16 at 15:37

The person who told you this is unit testing is completely wrong.

Unit tests are done, as you call it, at "source code level" (although I'd say that's a pretty vague description - after all, tests run code and only compiled code can be run, unless it's interpreted).

Unit tests rely heavily on mocking, that is, faking out components (dependencies) of a component (e.g. a single class) that you want to test.

What you are talking about in your question is something I call end-to-end testing. It's not an integration test either because those are still done without using an external program. Both integration and unit tests are usually performed using a specific test framework and test runner for the platform at hand.

Other terms that may apply for this case: system testing, acceptance testing (depending on what metrics are being validated).

| improve this answer | |
  • Thanks. does functional testing or blackbox-testing perform at executable level or source code level? – Tim Oct 30 '16 at 15:37
  • @Tim I don't know about the term "functional testing", but black box testing is a really broad term that just means "ignoring the internals". It can be a human clicking around in the GUI with the purpose of finding defects just as well. – MarioDS Oct 30 '16 at 15:41
  • does blackblox testing work on source code of the tested program, or on the executable of the tested program? Or it can be either case. – Tim Oct 30 '16 at 15:42
  • @Tim as soon as your test "knows" the source code (what's under the hood), it's not black box anymore. That is exactly what the term impies - no more, no less. It's antonym is white box testing. There is also a middle ground known as gray box testing. – MarioDS Oct 30 '16 at 15:45
  • 2
    Functional testing is about use case validations. They reproduce a real use case of the system. From the input data (by actors) up to the final (and expected) result. The way to perform these tests is, usually, at executable lvl. But they can be also implemented at source lvl. It depends on your QA protocol and resources. Functional testing can be automated at both lvls or executed manually. – Laiv Oct 30 '16 at 15:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.