6

Some background: I develop web services for internal departments of a large organisation that are used in public facing websites. There is a geographic differences between myself and my colleagues in these departments so most communication is by email, phone, skype etc.

My process for development is develop in test environment where only my team and I have access to it. This is fine and works well.

Once I and other members of my team are happy with the service I would upload and publish to a UAT/test environment which is accessible to the local domain (wider LAN area) for lets say eCommerce colleagues in another office to test their front end websites/applications against.

This is where the problem occurs. Testing would commence and then some time later eCommerce colleagues update their production/live environments to use this UAT service without my or my teams knowledge (obvious communication problem I know). I only find out when something has changed in UAT which breaks the service and eCommerce complain.

The services are clearly labelled with UAT in the titles/domain name. I have clearly specified when supplying them the UAT service not to use it in a live environment, keep me informed of testing etc. Then when all parties are happy with the service I would go through the relevant change control process and upload to the live production environment.

Are there any processes, methods, tips, advice I should be using to ensure UAT services are not used in a live environment that I have little control over?

  • Just a random thought: Would it be an option to periodically (every day, week, etc.) run a script that reverts all changes made by the users of the UAT environment? Losing the last X time of your work would be a big deterrent for putting the system in production. – Bart van Ingen Schenau Jan 13 '17 at 11:57
  • 2
    Why are they doing that? Do they not understand the difference? Are they frustrated by how long it takes the UAT to graduate? Do they have their own functionality that requires the new version? Was there a conversation about this last time it happened? With what outcome/agreement? – jonrsharpe Jan 13 '17 at 12:59
  • @jonrsharpe These colleagues are not technical. They understand the difference but probably not the process. They are not frustrated with the time just under pressure to update their front end with new functionality. Yes several conversations were had last time, but staff turnover in their department is quite high. Apart from communication issues is there anything technically I could adopt? – davidb Jan 13 '17 at 13:52
  • 2
    Sounds like a recipe for disaster that you have non-technical people integrating your services into their websites. All the good ideas below will be limited in their success utnil you rectify that situation – cdkMoose Jan 13 '17 at 18:13
  • 2
    I wouldn't focus on trying fix a people problem with a technical solution in this case. If a change in any non-production version breaks their code, that's not your fault or problem; every time they complain, point that out. If it's not actually causing problems for your development and testing, it would likely be productive to simply ignore it. – jonrsharpe Jan 13 '17 at 19:22
5

Davidb, I will try to dig down to what the problem actually is.

eCommerce colleagues update their production/live environments to use this UAT service without my or my teams knowledge (obvious communication problem I know).

There is no problem for you here. Why do you care what they are doing with their environment?

something has changed in UAT which breaks the service and eCommerce complain.

Probably this is the reason why you do care, right? Wait a moment. If the decision to use UAT services in production leads to problems on production then eCommerce department should make a complain to.... to whom? To those who made this decision - to eCommerce department! You are not involved in it at all.

The problem has solution which is not about technologies, you need to change your personal attitude. Don't accept any complains of that kind from eCommerce, you are not responsible for that.

4

In a perfect world, your code wouldn't know if it's in test or production. In that same perfect world, your colleagues would know better than to use a service for production before it is ready.

Since the world isn't perfect, need to make your service less appetizing until it is in production.

Some ideas:

  • As @BartvanIngenSchenau suggested in a comment, perhaps you can reset your database once a week. Since this is a testing environment, that would be perfectly reasonable and arguably necessary. By definition, there is no production data in a test environment so you are within your rights to do as you wish.

  • Consider shutting your service off when you are not actively testing it.

  • Have your service "watermark" its output. Append " TEST" to many of the strings you produce.

  • Speak with your Network Service people about possibly IP filtering access to the environment.

  • Resetting the database is not an option as in our scenario this is what is tested 100% of the time. The data needs to be in sync with production. Shutting the service down over night/weekends is probably the best option so far. – davidb Jan 13 '17 at 16:23
  • The watermark is a good solution. – ChuckCottrill Jan 14 '17 at 0:03
  • The UAT environment needs access to these services to test them, so ip filtering would not work. – ChuckCottrill Jan 14 '17 at 0:04
2

Here are a couple of options with some overlap to what Dan Pichelman suggested:

  • Network segmentation. Prevent calls from crossing between production and testing. This is also a good idea in the reverse direction for various reasons
  • Client certificates. If you use client-certs to authenticate, you will know which host is connecting to your service.

Ideally, you do both of these. The first is probably more reliable. The second works but if they can switch which servers are in their production environment, then they can game it. It would be a pain for them though especially when the next round of testing comes along. Client-certs should be fairly easy if you have an internal CA.

Really this is seems like a change management issue to me. That group should not be able to control the configuration of the production environment if they can't do it properly. If they are not doing it directly and someone is doing this on their request, you should contact that group and point out that by doing this they are violating policies that I presume you must have. There's all kinds of security issues with this. You can use that to scare people straight. Make them realize that if something goes wrong, it will come back to them.

  • Yes I agree with your comment. "this is seems like a change management issue to me" unfortunately I have little control or influence over that. I know my change procedures are up to standards but how do I control or influence others change procedures? – davidb Jan 13 '17 at 17:20
  • If there are policies in place and other teams are not following them, you can always rat them out to whomever it is that enforces those policies. I wouldn't start there personally because it could be perceived as an opening salvo in a war between your groups. Do you have any personal relationships with their team? Someone you have a little informal "hey what's with you not doing your TPS reports right?" conversation? If you can't do that or prefer not to, I would go to your management and explain the issue, the risks, and how it's somewhat out of your control. – JimmyJames Jan 13 '17 at 17:38
  • On a side note, you should still consider technical controls regardless of the fact that this is a people problem. They will help protect you from mistakes and bad actors. – JimmyJames Jan 13 '17 at 17:40
  • One more thought. You should also work with your own group to see what if anything you can do to make the move from UAT to Prod faster once accepted. Then at least when someone brings the hammer down on them, you can soften the blow. – JimmyJames Jan 13 '17 at 17:47
1

Firewalls and/or VLANs.

There should be a firewall between your production environment and your Dev/QA/UAT environments, such that a production service can't connect to a Dev/QA/UAT instance, for anything. Alternately, Production services are deployed on the Production VLAN, with no access to the Dev VLANs or QA VLANs.

Yeah, it can be a pain to work with, but it solves these sorts of problems by preventing them from occurring.

0

I think there is a general problem with release and deployment practices within your organization.

When your team develops a service, it will go through:

  • Build
  • Packaging

As part of Packaging a deployment Package is created. The package is then used to deploy to different environments. Each environment will have it's own specific configuration.

So, let's say you have package X which was created from a build from 1/13/2017.

  • Package X - UAT Config
  • Package X - PROD Config

One should never point PROD to point UAT. Promote the package (in this case X) to be deployed to production which will use the PROD configuration.

The configurations point to different URLs, database, etc to avoid crossing environmental boundaries.

Packages get promoted to environments, not an environment being pointed to a service.

  • 1
    Yes this is exactly what we currently adopt. The problem is the live environment which we have no control over is being point to UAT service which we do have control over. – davidb Jan 13 '17 at 17:17
  • That sounds like a process problem, which really isn't your problem. You could add a test only data element which production would not supply. Then all calls from PROD to UAT would fail. If you have control over UAT you could get some sort environmental value and check it prior to processing. – Jon Raynor Jan 13 '17 at 17:34
0

One choice might be to make an admin control panel, with appropriate authorization you could have a bunch of slider switches to turn on/off services with appropriate backend code to implement the accessibility.

0

Disclaimer: at my current employer we do continuous integration, so my advice is coming from that sort of environment. Hopefully there will be something you can use.

Have you considered more granular UAT? If each change undergoes UAT separately there will be a multitude of short-lived UAT environments, which necessitates differentiating their urls. We use a combination of the feature name and the instance ID, which is effectively random.

We've never had an issue with people hooking UAT code to the production database, but if someone got it into their head to do this, and somehow managed to get the permissions to make the required changes, finding a stable target would be impossible - when there's even a target for them to find.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.