How to stop UAT/QA/Test services being used in a production environment

Question

Some background: I develop web services for internal departments of a large organisation that are used in public facing websites. There is a geographic differences between myself and my colleagues in these departments so most communication is by email, phone, skype etc.

My process for development is develop in test environment where only my team and I have access to it. This is fine and works well.

Once I and other members of my team are happy with the service I would upload and publish to a UAT/test environment which is accessible to the local domain (wider LAN area) for lets say eCommerce colleagues in another office to test their front end websites/applications against.

This is where the problem occurs. Testing would commence and then some time later eCommerce colleagues update their production/live environments to use this UAT service without my or my teams knowledge (obvious communication problem I know). I only find out when something has changed in UAT which breaks the service and eCommerce complain.

The services are clearly labelled with UAT in the titles/domain name. I have clearly specified when supplying them the UAT service not to use it in a live environment, keep me informed of testing etc. Then when all parties are happy with the service I would go through the relevant change control process and upload to the live production environment.

Are there any processes, methods, tips, advice I should be using to ensure UAT services are not used in a live environment that I have little control over?

Answer 1

In a perfect world, your code wouldn't know if it's in test or production. In that same perfect world, your colleagues would know better than to use a service for production before it is ready.

Since the world isn't perfect, need to make your service less appetizing until it is in production.

Some ideas:

• As @BartvanIngenSchenau suggested in a comment, perhaps you can reset your database once a week. Since this is a testing environment, that would be perfectly reasonable and arguably necessary. By definition, there is no production data in a test environment so you are within your rights to do as you wish.

• Consider shutting your service off when you are not actively testing it.

• Have your service "watermark" its output. Append " TEST" to many of the strings you produce.

• Speak with your Network Service people about possibly IP filtering access to the environment.

Answer 2

Davidb, I will try to dig down to what the problem actually is.

eCommerce colleagues update their production/live environments to use this UAT service without my or my teams knowledge (obvious communication problem I know).

There is no problem for you here. Why do you care what they are doing with their environment?

something has changed in UAT which breaks the service and eCommerce complain.

Probably this is the reason why you do care, right? Wait a moment. If the decision to use UAT services in production leads to problems on production then eCommerce department should make a complain to.... to whom? To those who made this decision - to eCommerce department! You are not involved in it at all.

The problem has solution which is not about technologies, you need to change your personal attitude. Don't accept any complains of that kind from eCommerce, you are not responsible for that.

Answer 3

Firewalls and/or VLANs.

There should be a firewall between your production environment and your Dev/QA/UAT environments, such that a production service can't connect to a Dev/QA/UAT instance, for anything. Alternately, Production services are deployed on the Production VLAN, with no access to the Dev VLANs or QA VLANs.

Yeah, it can be a pain to work with, but it solves these sorts of problems by preventing them from occurring.

Answer 4

Here are a couple of options with some overlap to what Dan Pichelman suggested:

• Network segmentation. Prevent calls from crossing between production and testing. This is also a good idea in the reverse direction for various reasons
• Client certificates. If you use client-certs to authenticate, you will know which host is connecting to your service.

Ideally, you do both of these. The first is probably more reliable. The second works but if they can switch which servers are in their production environment, then they can game it. It would be a pain for them though especially when the next round of testing comes along. Client-certs should be fairly easy if you have an internal CA.

Really this is seems like a change management issue to me. That group should not be able to control the configuration of the production environment if they can't do it properly. If they are not doing it directly and someone is doing this on their request, you should contact that group and point out that by doing this they are violating policies that I presume you must have. There's all kinds of security issues with this. You can use that to scare people straight. Make them realize that if something goes wrong, it will come back to them.

Answer 5

I think there is a general problem with release and deployment practices within your organization.

When your team develops a service, it will go through:

• Build
• Packaging

As part of Packaging a deployment Package is created. The package is then used to deploy to different environments. Each environment will have it's own specific configuration.

So, let's say you have package X which was created from a build from 1/13/2017.

• Package X - UAT Config
• Package X - PROD Config

One should never point PROD to point UAT. Promote the package (in this case X) to be deployed to production which will use the PROD configuration.

The configurations point to different URLs, database, etc to avoid crossing environmental boundaries.

Packages get promoted to environments, not an environment being pointed to a service.

Answer 6

One choice might be to make an admin control panel, with appropriate authorization you could have a bunch of slider switches to turn on/off services with appropriate backend code to implement the accessibility.

Answer 7

Disclaimer: at my current employer we do continuous integration, so my advice is coming from that sort of environment. Hopefully there will be something you can use.

Have you considered more granular UAT? If each change undergoes UAT separately there will be a multitude of short-lived UAT environments, which necessitates differentiating their urls. We use a combination of the feature name and the instance ID, which is effectively random.

We've never had an issue with people hooking UAT code to the production database, but if someone got it into their head to do this, and somehow managed to get the permissions to make the required changes, finding a stable target would be impossible - when there's even a target for them to find.