1

We have a few apps that communicate with each other. I work on the core backend project X. X needs to make requests to a cloud native project Y in order to process some information.

Now, locally I don't run Y because it can't run locally since it depends on a lot of cloud provider specific stuff and I also don't have a dedicated instance of it since it costs a lot.

Instead we make requests to the staging version of Y even on localhost development. Localhost also needs to subscribe to the same messaging queue as staging does.

This means that me and each other developer as well as the staging environment for X all communicate with the same staging environment for Y and we need to filter through messages sent to the message queue to only read the ones we need. This is all leading to some issues.

My question is this: Is this a fundamentally flawed way of running our local environments or is this an approach that can be taken if done right?

EDIT As requested some of the issues I've encountered:

When something gets posted to a queue that localhost has to read, staging will also read from that queue and has to determine whether to discard it or if it's intended for elsewhere. Since some queue services don't allow filtering (e.g. Azure Service Bus queues) we will need to read the message in a non destructive way, check if it's related to our queue and delete it if necessary. If not we have to have a way to make sure we don't keep on reading that message again. These can be overcome by giving each environment its own queue but this increases complexity of service Y having to know which queue to send to.

Another issue is that a similar queue service (Azure Storage Queue) has a limit of how many messages you can read in 1 go, if say you can read 50 but localhost development environments have 20 messages in the queue already, you may only get 30 messages which lowers the efficiency making it harder to test performance improvements or stress test the system.

These can be overcome, but and maybe these are issues related to other design decisions but whilst working on it I was thinking that maybe the issue was the tight dependency that localhost had on a staging feature so I wanted to ask here to see what others thought.

3
  • 3
    Instead of thinking in categories of "okay" and "wrong", think in categories of "works for us" and "does not work for us". If your approach "leads to some issues" then it obviously does not work for you. So the real question would be "How can you make this situation work for you with acceptable effort"?
    – Philipp
    Feb 10, 2022 at 11:06
  • 1
    @KNejad, please edit your question to provide some of the “leading to some issues”. We can speak only to those issues only if we know what they are. Feb 10, 2022 at 15:15
  • Thanks @AndreasZUERCHER I added some information
    – KNejad
    Feb 10, 2022 at 18:12

3 Answers 3

4

Having local development environments communicate with a staging service (or in some cases even a production service) is not fundamentally flawed. There are situations where this is perfectly fine. For example, when the data is only being read and when there are no compliance issue with developers seeing the data or when the service is able to properly segregate testing data from production data.

But if it "leads to some issues" in your particular case, then it is flawed for you. So what are your options?

  • Pay for dedicated instances of the cloud service for development.
  • Implement a test double for the cloud service with an external interface which behaves in the same way as the real thing but without any of the backend functionality you don't need for testing your software during development.
  • Live with it.

Each of these approaches has a cost which can be quantified in money. The latter two don't seem to have an explicit price tag, but they do in form of developer hours which need to be paid for.

So which approach to choose is mostly depending on which one is the cheapest in your particular situation. We don't know anything about your service, so we can not tell you exactly how to quantify the cost and which one is the least expensive one.

2
  • 1
    Good thing to list the options. One note about quantifying costs: there sometimes are hidden costs that aren't easily quantifiable - for example, if dealing with the non-matching data can lead to hidden mistakes or log-term developer frustration, there may be future risks that aren't easily quantifiable now. Implementing a test double or mock of the real service may cost now but hopefully does not introduce hidden risks. Feb 10, 2022 at 13:09
  • 1
    @Hans-MartinMosner A self-developed test double bears other risks, though: Subtle implementation differences introduce bugs which only occur when run with the real thing. Integration tests in staging will hopefully catch this, but it would be better if such problems could be avoided during development.
    – Philipp
    Feb 10, 2022 at 13:10
0

The general advice is that everything belonging to X should be segregated per environment, to ensure that each version can be tested on its own.

However, Y is an external dependency, which means it's possible that there might only be a single instance to work with. A clear example here is an identity provider, which you might be using in all your environments.

As a second example, our project depends on another API hosted by another company. They have a production and staging environment. We use the staging environment for all non-production purposes (local, dev and staging) of our application.

Development environments should ideally be scoped only to the dev team of that project. External consumers (including other applications in the same company) shouldn't depend on it because (a) dev is a very volatile environment and (b) Y's dev team should not be hampered by coordinating with consumers in order to update their dev version.

That being said, if Y and X often release features together, then it can make sense to have the dev environments connect to each other, but this is IMO an indication that X and Y share a lifecycle, which begs the question if they are correctly separated.
There might be a good ulterior reason for their separation, but then the hassle of coordinating/breaking changes on the respective dev environments is the trade-off.

Ideally, I would keep dev private to the dev team, staging accessible for all non-production consumers, and production for production consumers. This ensures the most efficient development stages where you're minimally hamstrung by your consumers and any breaking changes they may suffer during your development cycle.

0

The staging infrastructure should be as representative as possible of the production environment with practically zero difference being the goal, so that the verification & validation testing in the staging area finds all the looked-for bugs, maldesigns, misconfigurations, etc that will show up after release to production. The more impaired that the staging area is to accomplish this goal of looking very close to production, the more bugs, maldesigns, misconfigurations, etc will be released to field production, where the penalty for these mistakes is much more severe. (If this isn't the primary purpose of staging, then why have staging at all?)

If the staging area is well-accomplished as doing a fine job of emulating the production environment, there is nothing wrong at all in having yet another copy/instance of the staging area deployed as the development infrastructure, such as for unit testing in isolation and for subsystem testing in isolation. In this way, the development infrastructure is minimally reusing the staging area as effort reduction (by development being a 2nd scripted deployment of the staging area's infrastructure), but not sharing/comingling/detracting-capacity-from the same instances.

What the original question describes though is a staging area that fails to closely approximate the production after release. What the original question describes is a development mock-up that somehow was reused (as even the same instances, which is double-bad) as a poor-man's staging area. The staging area should look toward the production environment for inspiration; the staging area should definitely not look toward a makeshift development mock-up as inspiration.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.