2

Currently, whenever I am developing a feature, I write the behaviour tests for it using mocks for our downstream services. These mocks are dumb always serve the same output. I faced something grave today. My BDDs were passing in my local and build environment. Therefore, I went ahead and merged my changes into the 'master' branch.

However, while testing the feature manually in my dev environment, I discovered a bug. One of the downstream services was responding with a 4xx. I know that this is bad - the master branch now contains a feature that is buggy.

If I would have written an integration test using actual services and made it a part of my build, this problem would have been solved. But say, if one of the downstreams goes down, then my build won't pass. I don't want this.

Note that we don't have a QA branch as we don't have a separate QA tester. We are all expected to test our features by ourselves.

How should I mitigate this issue?

  • In the absence of a QA tester, you simply have to do those things (like testing features manually) that give you a reasonable degree of confidence that your code works. There are no guarantees, even with QA testers, that your code is bug-free; and there isn't any automated thing you can put in that will catch every possible problem, because you can't know it's a problem until it happens. – Robert Harvey Jul 17 '19 at 21:39
  • Also, for what it's worth, there are some places (that have QA testers) which expect your work to be perfect before it ever gets to QA. The QA folks in those places will not help you at all. Sort of like having an HR department that is merely an advocate for the corporation. – Robert Harvey Jul 17 '19 at 21:40
  • Now that you understand the bug could you add a unit test that covers it? – candied_orange Jul 17 '19 at 22:18
3

Testing with stubs/mocks/fakes is good, but it can never fully replace system and integration tests where the real thing gets used.

If you don't have enough control over the external service to reliably use it in an automated integration test, then you should perform a manual integration test before you merge your work to the 'master' branch to prevent just the kind of embarrassment you got this time.

Another improvement to catch these error earlier (assuming the 4xx response was caused by a change you made in your code) is to make the mock for the external service smarter. Instead of always blindly serving the pre-configured data, the mock should also be able to check if the received request is as expected. If the request is not as expected, then there should be a failing testcase.

A test that validates that the correct request is sent still does not remove the need for an integration test with the real external service, because there can always be a different interpretation of the API between what you write in your testcase and what the real service implements.

| improve this answer | |
3

Not having separate QA is a very bad thing. At the most basic level, the software developer wants the software to work to demonstrate they did a good job, and the QA person wants the software to not work to demonstrate they did a good job. You can't do both at the same time.

| improve this answer | |
  • Explain that to my company, that believes develops should do everything. In this case however, can I add an integration test that I run locally but can disable during the build? What's the standard practice? – Prashant Pandey Jul 18 '19 at 4:25
  • 1
    "the QA person wants the software to not work to demonstrate they did a good job" - yuck. – Ant P Jul 18 '19 at 12:05
2

When you are writing mocks for the upstream services you depend on, you are matching the service's behavior for a limited number of cases.

If you get a 4xx error from the service in production, it means your mocks are not accurate to the service's actual behavior or the upstream service introduced a breaking change.

The best way around this is to checkout the services documentation and have your mocks more closely match it.

You can use something like OpenAPI Specification to document APIs.

| improve this answer | |
  • What if the services are not that pro-active in updating their documentation? I have faced this many a times. – Prashant Pandey Jul 18 '19 at 4:23
  • Since you are manually creating mocks for the service, you could write the documentation and generate the mock service from that, with something like apisprout – dcstone09 Jul 18 '19 at 18:19
1

If you are concerned about an integration test that interfaces with an unreliable external system, then one thing you can do is have a separate testing phase on your automated tests for the external test. This should not fail the whole build, if this phases fails (it may indicate a bug or it may indicate the downstream system has gone down), but indicates something needs investigation.

This requires a degree of discipline that you do investigate these issues and you must ensure your tests using your mocks are as good as possible, because you're not getting the same level of safety from external tests anymore, because they don't fail the build, but it can work reasonably well.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.