If you had a medium size company, several developers - but zero tests written in your REST API's - where would you start?

Question

I am looking to identify a benchmark to shoot for, and to come up with a decent game plan for introducing tests into our systems before deploying new code.

Just some backstory

My employer is not a tech company and our end users are inside the organization, we are just a department of 2-4 developers supporting a medium sized Corporation - but periodically the pain is really really felt when we roll out large software updates that span over multiple systems. The lack of tests usually leaves us fixing small issues that come up later for several days after a release, and makes us look bad sometimes (old features that people rely on break) after big wins (new features are introduced that people love).

My company uses a Microservice Architecture, every backend service is a stateless REST API we have about (12) backend services. We have (3) user facing websites written in several different languages that our company uses internally to do lots of their work in. The more and more we develop, the more and more people depend on our products, and the more and more it's going to be imperative that our services are working going forward.

In my mind, if we stopped everything we are doing and just wrote tests until I felt "safe" and that I had adequate testing for all of our infrastructure, it would take us up to 2 years to write them all. Thats just not something we can convince our co workers and executives to wait for.

--

If you were in our shoes, what sort of tests would you begin to write? Would you start with end to end testing? Like in each app, write tests to make sure that the app is able to query or update the other apps it communicates with?

Would you write tests inside something like Postman, and hit every API endpoint to see if its alive?

Or would you start internally for each app? Write things like unit tests for each function, or each REST API endpoint to ensure that each endpoint is working as expected?

there are so many places where work needs to be done, and its daunting to come up with a starting point where it would feel like we are making felt progress

Answer 1

I've actually been in this situation. When you talk about it taking 2 years to write the tests, I assume you mean unit tests. That's not where I would start.

What I (or my team, rather) have done around this is create a system of characterization tests (thanks Vincent Savard.) That is, you create a set of inputs, you input them into the system, and capture all the outputs. Then you take any modified version of the code and run these same inputs through, and compare the results. The main benefit of this is in a regression situation where you are making no functional changes to your application such as refactoring. In that situation, you can expect (with a few caveats) that the output will be the same for the same inputs which can be determined with standard diff tools.

You can do this kind of testing for functional changes as well. This a little more challenging to automate but it if you see things change in areas where you weren't expecting them, you can focus on that.

As mentioned, there are a few caveats: timestamps, UUIDs, etc. are not expected to be the same. Your comparison must be able to handle these and non-changes. Ignoring these fields is probably a decent first-cut but that does add risk. Ideally you should validate they they are there and fit the right format at the very least. This could be done as a separate step. Another issue that can come up is formatting and order. For example, some languages randomize hashing on each start of an application. If you use the default ordering from a hashtable, then you can get false positives. I've implemented sorting and formatting algorithms as part of the comparisons to get around this issue.

It's important to understand that when I say 'inputs' I mean all inputs which includes all state that can affect the result such as a database. Ideally, you create 'golden' datasets that you can restore to an exact state at the beginning of the test (or even during testing.)

Once you have this in place, you need to generate test suites. Random selections of production requests are often a good starting place but you want a sample that hits all the corner cases. One thing I haven't done which I think might be helpful is using code coverage tools to see how well your input sets exercise all the branches of the code. A more mature approach would have some curation of the test data to ensure it covers all the supported scenarios. You should consider error scenarios as well.

With this in place and a good amount of coverage, you can now confidently refactor. As as part of that refactoring, you can start introducing unit tests. Personally, I think unit tests are a little overrated and if you do this, I think you'll find that unit tests are really best reserved for your foundational/reusable components and that these regression suites are far more effective for everything else.

Answer 2

Five thoughts on priorities

1. Confidence is the main purpose of automated tests: To make you unafraid of changes and of frequent deployments.
   Write tests for areas in your code where you are worried.
2. Defect history: You aren't sure where you should be worried?
   Look at your last two years of defect fixes in your versioning system. If you see any modules that participate often, that's where you should be worried.
3. Unit tests are a waste of time for simple logic with high coupling, as is typical for most of your average web application.
   But a few(!) spots in such an application will have complex logic. Write some unit tests for those.
4. Smoke tests, in contrast, are a must-have: A rather small number of tests that cover a lot of ground in your code base. Stick to happy-path testing (no error cases) to keep them simple, but try to call each oft-used functionality once.
   These tests are a pain-in-the-ass to debug if they fail, but at least you know you've broken something. And after you've tracked that down, you will be very motivated to add a smaller-grain test for next time.
5. UI-based tests are also a pain, because they keep breaking for irrelevant reasons.
   So try to keep your large-grain tests one level below the UI level if you can. REST APIs are a good basis for that.

If you follow these priorities stringently, you can well leave everything else for when you have time (aka never).

Answer 3

This story begins where it ends: the testing pyramid:

Source: https://www.ministryoftesting.com/dojo/lessons/the-mobile-test-pyramid

You do not want your initial steps to compromise where you need to be. Unit tests should comprise the majority of tests. This may require different programming techniques or refactoring that is prohibitive right now, because it could introduce more problems than it fixes. Instead, focus on the top of the pyramid first.

Your initial set of tests should cover the most critical use cases. These end-to-end tests can use any tool you like. The important part is that these tests should entirely interact with the user interface of the application. Focus on a small number of the most crucial acceptance tests. Consult a domain expert in each of the applications you support to help determine those use cases. For instance, when creating a blog post, write one test that captures a typical situation where a user enters a sensible title, sensible body text and adds a few tags to categorize the new post. Build out the very tip of the pyramid first, and then stop writing new end-to-end tests for old code.

Ideally the e2e tests should be something you can execute against any application environment. If production is sensitive enough due to personally identifiable information or financial transactions, then aim to execute these e2e tests against a local environment, dev and test. Set up Continuous Integration to run theses tests whenever code is merged into your main version control branches.

End-to-end tests are brittle and have a myriad reasons to fail. When they fail, the root cause will not be obvious, but this is a compromise so the failures are encountered before code is deployed to production.

In parallel, all new features should ship with new automated acceptance tests. All new bug fixes should be covered by e2e tests as well, but just enough to cover the bug. These new tests should be included in your continuous integration builds. This is a good time to introduce unit testing for new code, which could include introducing things like dependency injection in selective areas, where this would be easy and low risk.

Once you're team has accomplished this, focus on the next layer down in the pyramid: integration tests.

Here again, the critical use cases identified by the domain experts for each application should determine which components get integration tests first. These should also be included in your continuous integration builds as well. Integration tests should run first, then the e2e tests.

In parallel, all new features should be shipped with integration tests and automated acceptance tests. If possible, all new bug fixes should be shipped with integration tests. Certainly keep covering bugs with e2e tests. Keep trying to increase unit test coverage for new features. Budget some time to selectively add unit tests for old code where it is easy and low risk.

At this point the extra effort to write these tests should be offset by the reduced time spent bug squashing. Now you can focus on refactoring code to make unit testing easier. This phase includes the bigger, more risky refactoring jobs so your test pyramid starts to look right. Unit test coverage should accelerate dramatically during this period, and then make an effort to keep this up.

Answer 4

Its a classic.

To completly "clean up" the system would take to long, because there are people in dire need of new functionalities. But without a clean up new functionalities need longer to implement (and break quite often).

A vicious circle.

One thing i would start with, is that i would change the implementation process. Before, functionalities were implemented without automatied tests. Now each change may only be delivered if there is a good enough test coverage. Yep, it can still be that old parts will break because of new changes. And that those are not tested. But the precise wording is "not tested yet". Because when those parts break, then you will have to fix it, that means changing the code and that means adding tests.

As a result over time the brittle parts of your system(s) will get more and more test coverage. And therefore get safer. The parts that do not break very likely, will have no or only a few tests. But thats okay, because it seems they are not that much at risk.

Its very hard to provide proper automated tests when you have to deliver hotfixes to the production asap. Therefore if it is not possible to deliver them AFTER creating and running the tests, make at least sure that all new functionalities are postponed until those tests are delivered.

I love UnitTests, but primarily because Test Driven Development really enforces me to think about the design of the code. In your case it may help to start using them, but only as a helper for new development. Writing UnitTests for existing code which was not designed with UnitTests in the back of the mind is a really horrible task.

Therefore i would also use the approach mentioned by @JimmyJames. I would define a set of testing inputs for a single system record the output and then compare the results before and after a code change. This is done for each single system.

I would try to not use that approach for a system of systems. In generell then the outputs get very complex and it takes long to define a goot set of inputs and collect all relevant outputs. Therefore that effort i would only invest if multiple times my system tests were all green but still the overall system runs into problems.

Answer 5

Do work that is useful. Since the company lived without unit tests, you’ll have to convince management of actual financial benefits first. This can go well or bad for your reputation

If your app works reliably, all tests for existing code should pass any new unit tests that you write, unless your testing is bad (say you have code that should do X, but does Y which is different, but plausible enough that QA never spotted the difference.)

So first write a framework with one unit test to have a pattern people can follow. Then you add unit tests (a) when you make a change to test that behaviour before and after the test is the same) and (b) for new functionality.

So don’t write unit tests just for the sake of it, because that will be just cost without benefit.