Should every service have its staging environment?

Question

I'm trying to break down my dockerized monolith into a few services that can be deployed independently.

Right now, I have a development environment, a staging environment and a production environment. I'm not sure how it's supposed to work for microservices, especially for the staging environment. Is every service supposed to have its staging environment, which communicates with its staging counterparts? Same from the development environment, should it use the production services?

I've tried to do research on the subject, but there doesn't seem to have a clear answer. At least I didn't find one.

Answer 1

If you go down this route then you start needing multiple permutations of environments.

Instead I would suggest you do versioned APIs with multiple versions live at the same time.

Use header based routing to do this and you can release and upgrade parts of your system while keeping other parts on older versions.

You can have the latest release as a beta and limit access if required.

Answer 2

When you promote the recommendation service from staging to production, do you want it to crash because the production version of the user accounting service is older than the one it was tested against?

I thought not. Therefore, you must test it against the same versions of other services that it will be deployed against, which are the ones currently in production.

Answer 3

I get the feeling you are about to fall into infrastructure oversizing. Your doubts are reasonable, but they also say that you don't have any fact that you need such environment segmentation. At least not yet.

What do you have so far? You have a monolith, which is developed and operated across three environments. They are already provisioned with everything the solution needs to be functional. I can't think of a better starting point to test (and compare) the breakdown of the monolith.

As more services appear in the ecosystem, the more likely you need the flexibility to test different permutations of services. That flexibility is achieved by configuration and parameterization.

Bear also in mind that environments are not mere infrastructure segmentations. Environments are composed of runtime and data. Data on production is often subject to regulatory compliance, so you don't assume easily that "development should use the production services". It's not that simple.

Think also of the costs. If you were to deploy production on some of the well-known public cloud platforms, you would realize how using production services for testing development can wipe your budget in a breath.

Another reason to avoid mixing up environments is metrics. Cross-env resource utilization can invalidate or distort the lecture. It's going to be hard to monitor Production for adaptions if random activity in Development (or other envs) is generating "noise".

As an MS architect, your mind should always be on the big picture. That involves the software but also infrastructure, security, operations, cost-optimization, regulatory compliances, tracking, metrics, business goals, etc.

Answer 4

It really boils down to your approach to the developer or team organization and the release methodology.

The snapshot (collaborative) approach:

1. The whole product (the whole bunch of micro-services, some with new changes, some same as the old) is considered a change "snapshot".

2. On every change, may be it in one micro-service, or many, the snapshot of whole product is deployed to a dynamically generated "preview" environment for that feature or change. Changes worth releasing eventually make it out of here to integration.

3. There is one integration environment where these snapshots are merged. This is not code merge, but, integrating micro-service versions as they come. For Example:

   Team1/Developer1 patched service 1, Snapshot S1(service1:v1.1, service2:v1.0)
   Team2/Developer2 patched service 2, Snapshot S2(service1:v1.0, service2:v1.1)
   
   Integration environment RC1 = apply S2 over S1 = (service1:v1.1, service2:v1.1)
   

   Here are all the end to end tests that make sure the versions of micro-services work well as a whole product. Only passed version combinations make it to the next stages, these are the stable, release candidate "snapshots" of the product.

4. From here on, it is up to you to promote the stable product snapshot between as many environments you like, staging, beta, production, whatever.

5. Finally, you can release the snapshot of product. Usually, only the services with newer version in the snapshot are restarted.

This approach is suitable where in all micro-services are somewhat tightly coupled and are developed in close collaboration.

The patch (isolated) approach:

1. Micro-services are patched, tested and released individually. Stability ensured beforehand.
2. Each change to a micro-service is deployed to a dynamically generated "preview" environment, with rest-of the environment being a replica of production versions.
3. The individual service development is kind of "isolated" from what's going on in the other services development, its only reference of them being the production versions.
4. This makes the "preview" environment here, also an "integration" with "production", hence integration & end to end tests can approve of this one service's compatibility with production here itself.
5. And here comes the catch: Release to production can only be one service at a time. Each release changes "production". The other services being worked on, have their "preview" environments reflect this new production, and have to reconcile and be compatible with it before making the next attempt at release.

This approach is suitable where micro-services are loosely coupled and are developed by isolated developers or teams.