I am looking to identify a benchmark to shoot for, and to come up with a decent game plan for introducing tests into our systems before deploying new code.
Just some backstory
My employer is not a tech company and our end users are inside the organization, we are just a department of 2-4 developers supporting a medium sized Corporation - but periodically the pain is really really felt when we roll out large software updates that span over multiple systems. The lack of tests usually leaves us fixing small issues that come up later for several days after a release, and makes us look bad sometimes (old features that people rely on break) after big wins (new features are introduced that people love).
My company uses a Microservice Architecture, every backend service is a stateless REST API we have about (12) backend services. We have (3) user facing websites written in several different languages that our company uses internally to do lots of their work in. The more and more we develop, the more and more people depend on our products, and the more and more it's going to be imperative that our services are working going forward.
In my mind, if we stopped everything we are doing and just wrote tests until I felt "safe" and that I had adequate testing for all of our infrastructure, it would take us up to 2 years to write them all. Thats just not something we can convince our co workers and executives to wait for.
If you were in our shoes, what sort of tests would you begin to write? Would you start with end to end testing? Like in each app, write tests to make sure that the app is able to query or update the other apps it communicates with?
Would you write tests inside something like Postman, and hit every API endpoint to see if its alive?
Or would you start internally for each app? Write things like unit tests for each function, or each REST API endpoint to ensure that each endpoint is working as expected?
there are so many places where work needs to be done, and its daunting to come up with a starting point where it would feel like we are making felt progress