0

Firstly, let me clearly state that I am completely against this, I have real issues with unnecessary, obsolete, untested, shitty code, especially when they end up on a production system. My background is in build and release management/devops and I am now working as a Technical Consultant supporting an outsourced development team.

The scenario is this;

We have an MI system that published data from approximately 200 source tables into 200 destination tables by ways of a single 'dynamic' stored procedure that generates and executes MERGE statements (for each table). The content of these MERGE statements are driven by various configuration tables (source to destination at the column level).

While I am not a complete fan of this way of doing things, it has been in place since release 1 (years) and it works and is well tested, and surprisingly well written given what/how it does.

There is a separate project that has a goal of reviewing the system as a whole and making performance improvements where possible. Absolutely no issue with the goal here.

As part of this project, there has been a discussion that perhaps the dynamic MERGE statement might be more performant as a static stored procedure. Perhaps there could be improvements by caching execution details, etc. I am not a DBA, but the general idea sounds feasible. We can do an in-depth analysis of each table, and tune if possible (add additional indexes).

While converting some of the dynamic elements might make sense, perhaps this is not suited for all and adding the artefacts and converting all to static is an overhead that might not be necessary. So in order to support this, we discussed an extension to the existing configuration framework that allows a switch between dynamic and static. All good here so far.

We then go into the subject of testing, and validation of the performance at scale. Generally the approach to this was fine, but there was going to be a delay in getting a suitable environment, with a suitable set of data ( we are talking billions or rows/terabytes of data)

I have recently found out that the intention is to implement the extension to the configuration framework and push this live, with no real testing, and no validation of the performance gains. The icing on the cake to me is that it will not actually be enabled.

Our deployment process is currently fairly complex, and releases to production are done in a standard manner regardless of the payload. There is a release scheduled in a few months and these changes will go live. The next release has no go-live schedule. All changes such as this are done as a DACPAC deployment, so whether there are changes or not, the process will compare the DACPAC to production for each component.

So between this release and next, these changes will exist, untested, but unused, while the analysis is performed. Once complete, the next release will update the configuration data and any supporting artefacts.

I appear to be fighting a losing battle with regards to convincing the (technical) project manager of the third party that this is not a good approach and have simply asked why it could not be released once the analysis has been done, in the next release (if applicable still)

I am generally curious as to whether I am being too subjective about this and its not really a big issue. I have just sent another email to a wide circle of people clearing stating my concerns and it will be interesting to see the response, I am guessing it will be along the lines of we are doing this anyway.

Any thoughts or comments?

Thanks

  • 1
  • 3
    also, "Questions require a goal that we can address. Rather than explaining the difficulties of your situation, explain what you want to do to make it better. For more information, see this meta post..." – gnat Jul 19 '17 at 9:07
  • Please ask a specific question. Are you trying to find what the BKM is for evaluating performance improvements made to a system that is constantly in use? – Jay Elston Jul 20 '17 at 5:02
  • @JayElston I think my question is specific enough, others have managed to decipher it and offer excellent responses. I am sorry that it does not fulfil the requirements of a perfect question, but sometimes people are after general thoughts. This particular stack exchange site does not have a requirement, but a preference for questions that can be 'answered', so in my opinion it's good to go. Thanks for you input anyway. – ChrisBint Jul 20 '17 at 7:55
  • @gnat apologies that I did not have the time to peruse the blog relating to stackoverflow, a different site in the stack exchange (which has a clearly defined set of requirements for a question, which mine did not fit, so I did not post it there). The same as I was unable to find the time to go through the workplace stack exchange site/meta before posting. Thanks for the first link however, that was useful. – ChrisBint Jul 20 '17 at 7:58
3

Your exact technical solution and implementation seem somewhat niche. I did however work for a company where putting something live in a turned off state was common. In our case it was a winforms application with a flag in the database or config file that could turn the feature off and on, but I'm hoping some of my experiences can assist with the problems you're seeing here.

THE GAIN

Once the code is put live, even if it isn't used, it is a tick in a box and the deployment has at least been proven. This however, is the only upside.

THE PAIN

Minor revisions break the new functionality

Once the big change goes in, sometimes even the simplest subsequent change can cause this to unravel. Because the big change isn't actually live, it often gets missed in the testing or isn't considered in later designs.

YAGNI

You shouldn't be writing code unless you need it. If you need it, you should deploy it. If you've delayed turning the new code on, new requirements may have come to light meaning that the code could and should have been written better.

Technical debt

This in some ways is the worst outcome. The new code is never turned on. Bigger priorities come in and the new code is parked. It is too much of a risk to back it out and so it remains in the deployment, clogging up the code base and hogging resource.

Performance

Even though the new feature isn't visible. Sometimes related processing still takes place behind the scenes meaning the application degrades. As far as the user goes this is lose-lose. There is no new functionality and yet the system is performing slower.

The switchover fails

Due to minor subsequent changes or unforeseen circumstances on the live system, sometimes the switchover simply fails. The best scenario here is that you can simply turn it off again but if data is affected you could have a major issue on your hands.

SUMMARY

As with all things of this ilk, if it comes off - it looks like sheer brilliance. The release has gone in ahead of time and a magic switch makes the new functionality just work. But it rarely happens like this. Even if it does, the rare failures should immediately set alarm bells ringing that this is poor deployment practice.

  • Thanks for the input :) Out of curiosity, in your case was the feature at least tested, so the flag could be enabled with confidence? – ChrisBint Jul 19 '17 at 10:12
  • In most cases it was tested against the system as it was at the time, but the longer the deployment is out there without being switched on, the more likely it is that minor deployments and things like data anomalies can cause the feature to break. It is a leap into the unknown really... – Robbie Dee Jul 19 '17 at 10:15
1

... single 'dynamic' stored procedure that generates and executes MERGE statements ...

... dynamic MERGE statement might be more performant ...

... the intention is to ... push this live, with no real testing, and no validation of the performance gains.

Question: Does it matter if this untested "alternative" messes up the data or runs many times slower than the current method?

I doubt either of these would be acceptable and either/both would become evident in Testing.

So between this release and next, these changes will exist, untested, but unused, while the analysis is performed. Once complete, the next release will update the configuration data and any supporting artefacts.

It will not be "unused".

It will be executed, in production and updating data as it does so, in order to work out whether it works and, assuming that it does, whether it's any better/worse than the original.

I appear to be fighting a losing battle with regards to convincing the (technical) project manager of the third party that this is not a good approach and have simply asked why it could not be released once the analysis has been done, in the next release (if applicable still)

Or, in other words, why can't it be tested prior to being put into production?

They might argue that the size of data you're working precludes the use of a testing environment, but that would also suggest that your testing environments are not representative of Production, and that's another problem.

0

My thoughts are nothing should ever be sent into production without a barrage of testing, even standard unit testing. This is no different. It's precisely that assumption that nothing will go wrong that creates spaces where bugs can squeeze inside. Even if there are no bugs, repeated instances like this only means it's a question of when not if.

Perhaps you can't prevent this from going into production, but you can put your foot down to ensure that this gets tested assuming your technical advice has any weight in the matter.

Better still, this development should be in a branch until the day in which this will actually be used in production so that there would be no impact should the changes never be actually required. And of course if and when the changes do need to be brought to production, the changes should be integrated into the trunk, tested thoroughly once more both in disabled and enabled states, pushed to production disabled and then enabled at a convenient moment. This won't eliminate the possibility of bugs entirely, but it should certainly minimize problems.

0

I largely agree with my colleagues Nothing should ever be put into production without having been thoroughly tested, however that's the idealist in me talking, the realist side of me says it happens and it happens a lot. there is risk that if the code isn't commented out correctly you could get production issues, but the gain is that the release can go ahead quickly without reams of regression testing. However again in taking this approach you are filling the database with lots of redundant code which may never be used this is not good practice.

I don't think you are over egging the situation and I agree with your principles but I think you need to arrive at a pragmatic agreement on the way forward.

  • This really is not an answer, but the OP did not really ask a question, either. This site is not a good forum for discussion such as "look at how poor SW process is at my place of employment" / "I agree it sucks". – Jay Elston Jul 20 '17 at 5:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.