-6

First, the most obvious grouse someone has against this I can think of is the intricacies of an actual method. It's not enough to merely ensure no errors are thrown. Functions usually contain alternate paths borne of if/else constructs. For this, assume I use a declarative syntax that causes the generated/in-memory tests to follow all those paths. Also assume test framework can detect valid input from the endpoint's validation rules.

Another common test expectation is during unit tests, that certain methods return specific values. Or, during integration tests, that interacting with elements on a page matches asserted values.

Here are my questions:

  1. why isn't it sufficient to know method executed successfully, or that the return type conforms to method's type-hint?
  2. Is it really scalable to edit our tests each time an expected specific in the implementation is replaced?
  3. Why doesn't integration tests involve the above described procedure replicated for each of the other endpoints part of that feature?

Apparently, the inventor of PHPUnit itself embarked on a similar undertaking some years back, but unfortunately deserted it. What challenges could be behind his resignation of defeat? This SO answer hints thus

it's also an endless long-term project

I don't have enough forum rep to ask what could possibly protract its completion. Why is this seen as such a dangerous form of securing a codebase?

EDITED

I disagree with the comments by @jonrsharpe and @user253751, and this is my defense. I am only interested in testing action methods tied to all defined routes. I have a test case for /is-prime-number/5. The handler calls an internal/private method that decides whether or not a number is a prime number. Why do I have to redefine another method within the test case that determines whether or not its main counterpart is correct? If I got the SUT calculation confirming a number is a prime number wrong, what's the guarantee that the test case calculation will be correct? The only way this can be avoided is if it's another developer writing the tests for someone else's code, which in my experience, is rarely the case.

The code already knows what it wants to do. Tests ought to confirm it successfully executes its desires both during runtime and at compile time. Such specific domain specific semantics as whether the number is a prime number in real life should exist within the scope of the business logic itself.

EDIT 2

Users on this board are downvoting me because they incorrectly think I'm referring to functional testing. I want tests that insulate updates from creating regression bugs. The contents of the response have been manually tested via the browser/console/postman/network tab etc. and are certainly satisfactory to me before I'm ready to commit/run tests

15
  • 4
    If by automating automated tests you mean generating tests for existing code, the fundamental problem is the tests don't know what the code is supposed to do. All they can do is ensure that the code keeps doing the thing it currently does. Is that the right thing? There's no way to be sure. Is it possible to refactor confidently? Not really, because they're based on the implementation not the behaviour. What do you do when tests fail because you changed something? Just click "regenerate"? How do you tell whether that failure was useful information? – jonrsharpe Nov 13 '20 at 15:21
  • 2
    Your question does not make enough sense hence all the downvotes. You are going to have to expand on what you mean by 'automating your automated tests'. Are you hinting at automatically generating them? – c_maker Nov 13 '20 at 15:21
  • 2
    "The code already knows what it wants to do." Great news! Then you don't need tests at all, and there are no bugs. What you're describing is change detection, it doesn't help you in any meaningful way. Beyond that, I'm not sure what you're trying to say. – jonrsharpe Nov 13 '20 at 15:55
  • 2
    I mean a test failure just tells you whether or not the SUT has changed. It doesn't tell you whether the behaviour before was right, or whether the behaviour after (still) is. "matches my manual test" - so you already have test cases, automate those. "It's up to the business guys/product owner to complain..." - that seems like quite an unprofessional attitude, take some responsibility for the value of your work. – jonrsharpe Nov 13 '20 at 16:05
  • 1
    "I manually confirm they return expected values when I reload the browser/console/postman" - so automate that. You don't need to automatically generate test cases, those are your test cases. Use tools like Selenium to interact via the browser in the same way you have manually, make those tests faster and more repeatable than doing it by hand. – jonrsharpe Nov 13 '20 at 17:06
3

It's pretty simple: You want to know that the function does the right thing, not just that it returns the correct type.

bool isPrimeNumber(int n) {
    return true;
}

If your test says this function is okay (because it executes successfully and returns the correct type) then your test is useless.

6
  • Also if it doesn't test primes and non-primes, it's no better. – Deduplicator Nov 13 '20 at 15:37
  • I replied you in my edited answer – I Want Answers Nov 13 '20 at 15:52
  • So, according to your responses, the test generator is safe if I can define the expected outcome for each test (simply with an assert)? – I Want Answers Nov 13 '20 at 16:27
  • 5
    @IWantAnswers That's called writing tests! – user253751 Nov 13 '20 at 16:58
  • What if the declarative control flow equally provides an interface for specifying expected outcome? Will I be crucified for violating single responsibility principle? Is it forgivable if converging both utilities at the same location solves 1) people learning to write tests 2) time to write it 3) makes tests unavoidable 4) updating along with the updates to the code – I Want Answers Nov 14 '20 at 16:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.