Pushing untested, unproven, unused code to production with a view of future use

Question

Firstly, let me clearly state that I am completely against this, I have real issues with unnecessary, obsolete, untested, shitty code, especially when they end up on a production system. My background is in build and release management/devops and I am now working as a Technical Consultant supporting an outsourced development team.

The scenario is this;

We have an MI system that published data from approximately 200 source tables into 200 destination tables by ways of a single 'dynamic' stored procedure that generates and executes MERGE statements (for each table). The content of these MERGE statements are driven by various configuration tables (source to destination at the column level).

While I am not a complete fan of this way of doing things, it has been in place since release 1 (years) and it works and is well tested, and surprisingly well written given what/how it does.

There is a separate project that has a goal of reviewing the system as a whole and making performance improvements where possible. Absolutely no issue with the goal here.

As part of this project, there has been a discussion that perhaps the dynamic MERGE statement might be more performant as a static stored procedure. Perhaps there could be improvements by caching execution details, etc. I am not a DBA, but the general idea sounds feasible. We can do an in-depth analysis of each table, and tune if possible (add additional indexes).

While converting some of the dynamic elements might make sense, perhaps this is not suited for all and adding the artefacts and converting all to static is an overhead that might not be necessary. So in order to support this, we discussed an extension to the existing configuration framework that allows a switch between dynamic and static. All good here so far.

We then go into the subject of testing, and validation of the performance at scale. Generally the approach to this was fine, but there was going to be a delay in getting a suitable environment, with a suitable set of data ( we are talking billions or rows/terabytes of data)

I have recently found out that the intention is to implement the extension to the configuration framework and push this live, with no real testing, and no validation of the performance gains. The icing on the cake to me is that it will not actually be enabled.

Our deployment process is currently fairly complex, and releases to production are done in a standard manner regardless of the payload. There is a release scheduled in a few months and these changes will go live. The next release has no go-live schedule. All changes such as this are done as a DACPAC deployment, so whether there are changes or not, the process will compare the DACPAC to production for each component.

So between this release and next, these changes will exist, untested, but unused, while the analysis is performed. Once complete, the next release will update the configuration data and any supporting artefacts.

I appear to be fighting a losing battle with regards to convincing the (technical) project manager of the third party that this is not a good approach and have simply asked why it could not be released once the analysis has been done, in the next release (if applicable still)

I am generally curious as to whether I am being too subjective about this and its not really a big issue. I have just sent another email to a wide circle of people clearing stating my concerns and it will be interesting to see the response, I am guessing it will be along the lines of we are doing this anyway.

Any thoughts or comments?

Thanks

Answer 1

Your exact technical solution and implementation seem somewhat niche. I did however work for a company where putting something live in a turned off state was common. In our case it was a winforms application with a flag in the database or config file that could turn the feature off and on, but I'm hoping some of my experiences can assist with the problems you're seeing here.

THE GAIN

Once the code is put live, even if it isn't used, it is a tick in a box and the deployment has at least been proven. This however, is the only upside.

THE PAIN

Minor revisions break the new functionality

Once the big change goes in, sometimes even the simplest subsequent change can cause this to unravel. Because the big change isn't actually live, it often gets missed in the testing or isn't considered in later designs.

YAGNI

You shouldn't be writing code unless you need it. If you need it, you should deploy it. If you've delayed turning the new code on, new requirements may have come to light meaning that the code could and should have been written better.

Technical debt

This in some ways is the worst outcome. The new code is never turned on. Bigger priorities come in and the new code is parked. It is too much of a risk to back it out and so it remains in the deployment, clogging up the code base and hogging resource.

Performance

Even though the new feature isn't visible. Sometimes related processing still takes place behind the scenes meaning the application degrades. As far as the user goes this is lose-lose. There is no new functionality and yet the system is performing slower.

The switchover fails

Due to minor subsequent changes or unforeseen circumstances on the live system, sometimes the switchover simply fails. The best scenario here is that you can simply turn it off again but if data is affected you could have a major issue on your hands.

SUMMARY

As with all things of this ilk, if it comes off - it looks like sheer brilliance. The release has gone in ahead of time and a magic switch makes the new functionality just work. But it rarely happens like this. Even if it does, the rare failures should immediately set alarm bells ringing that this is poor deployment practice.

Answer 2

... single 'dynamic' stored procedure that generates and executes MERGE statements ...

... dynamic MERGE statement might be more performant ...

... the intention is to ... push this live, with no real testing, and no validation of the performance gains.

Question: Does it matter if this untested "alternative" messes up the data or runs many times slower than the current method?

I doubt either of these would be acceptable and either/both would become evident in Testing.

So between this release and next, these changes will exist, untested, but unused, while the analysis is performed. Once complete, the next release will update the configuration data and any supporting artefacts.

It will not be "unused".

It will be executed, in production and updating data as it does so, in order to work out whether it works and, assuming that it does, whether it's any better/worse than the original.

I appear to be fighting a losing battle with regards to convincing the (technical) project manager of the third party that this is not a good approach and have simply asked why it could not be released once the analysis has been done, in the next release (if applicable still)

Or, in other words, why can't it be tested prior to being put into production?

They might argue that the size of data you're working precludes the use of a testing environment, but that would also suggest that your testing environments are not representative of Production, and that's another problem.

Answer 3

My thoughts are nothing should ever be sent into production without a barrage of testing, even standard unit testing. This is no different. It's precisely that assumption that nothing will go wrong that creates spaces where bugs can squeeze inside. Even if there are no bugs, repeated instances like this only means it's a question of when not if.

Perhaps you can't prevent this from going into production, but you can put your foot down to ensure that this gets tested assuming your technical advice has any weight in the matter.

Better still, this development should be in a branch until the day in which this will actually be used in production so that there would be no impact should the changes never be actually required. And of course if and when the changes do need to be brought to production, the changes should be integrated into the trunk, tested thoroughly once more both in disabled and enabled states, pushed to production disabled and then enabled at a convenient moment. This won't eliminate the possibility of bugs entirely, but it should certainly minimize problems.

Answer 4

I largely agree with my colleagues Nothing should ever be put into production without having been thoroughly tested, however that's the idealist in me talking, the realist side of me says it happens and it happens a lot. there is risk that if the code isn't commented out correctly you could get production issues, but the gain is that the release can go ahead quickly without reams of regression testing. However again in taking this approach you are filling the database with lots of redundant code which may never be used this is not good practice.

I don't think you are over egging the situation and I agree with your principles but I think you need to arrive at a pragmatic agreement on the way forward.