5

What are the general strategies to employ when attempting to ensure that a module of code will function correctly on the live system? A common problem we have at our software house is that we typically deploy solutions to a remote client's server, which hasn't got any software-development-related software installed on it. When problems (inevitably) arise in the deployed software, we get annoyed clients telling us to fix the issue, but since we can't remotely debug the code, we get locked into a cycle of effectively having to guess whether our code will work before we deploy it, and then repeating the cycle when the deployed code has issues.

Furthermore, due to the remoteness of the client, it's typically not possible to get a full working test copy of their system set up on our end, so we end up developing against approximations of their system on our end. This makes our debugging and code planning on the development side significantly less effective, because when we deploy we don't know precisely what variable between our test system and the live system is causing problems with our code.

So far, I've implemented a basic error logger in all deployed code that captures the stack trace, exception details, method, class, namespace, parameters and additional messages and error codes at each exception, but this really just speeds up the process of fixing that specific error. I've also tried writing individual tests for specific subsections of modules as executables and running them on the client system, but for modules that are tens of thousands of lines long, this usually just isn't feasible without grinding all development to a halt.

I'm trying to get a strategy in place which would help us better avoid the exceptions in the first place, but I'm stuck on the fact that:

  • We don't have (and typically can't get) a working test system that accurately models the live system, e.g. an image of the live system, if we do have it then it's months out of date

  • We can't install remote debugging software on the live server

  • The client typically has no dedicated test server, so any deployments have to go straight to live

  • Having no access to a copy of the live system means that we can't write effective unit tests which actually model the conditions the code will run under on the client side

If it helps, the code is typically C# running under .NET 3.5. How to better tackle this issue?

Improve this question

2 Answers 2

Reset to default
11

We don't have (and typically can't get) a working test system that accurately models the live system, e.g. an image of the live system, if we do have it then it's months out of date

This is the problem. You know this is the problem.

The more complex a system, the more integration points it might have, the more likely you are to have issues.

Why can't you get a working test system? Will your company or client not pay for the servers? Is the data typically sensitive? Can it not be anonymised?

Do you have testers? Why not? Will your company not pay for them?

Typically these types of things should be costed into the build of a system, but as often happens, price is always an issue for clients. You get what you pay for as they say.

Do you get enough time to get all your unit tests in place?

As a programmer at the coal face, you can only mitigate these things so far.

Having your unit tests and code coverage as high and up-to-date as possible, and having your system using Dependency Injection to help facilitate testing by effectively mocking out integration points is a great start.

If a manager comes moaning to you because something has failed in live, this list of reasons is a good place to start as to why it is so difficult to deploy under these conditions.

Improve this answer
7
  • Yeah, I agree that's the issue. Management are entirely non-technical, and they make promises to clients without doing basic feasibility. They basically don't bother to contact the dev team until after they've promised something they can't implement, and the opportunity for telling the client that we need a test system is long gone. Typically, clients don't have a test system in place because they don't care about IT. Getting an image of their system is typically infeasible because of poor internet connections and remoteness of clients.
    – Ben H
    Commented Mar 27, 2013 at 11:48
  • 2
    @BenH: "clients don't have a test system in place" - of course, that is your companies responsibility. "Getting an image of their system is typically infeasible because of poor internet connections" - write your client a program which they can use to create a dump of their live system / database into a file etc. and let them burn it to disk. Let them send that disk to you by FedEX. Worked for us some years ago very well.
    – Doc Brown
    Commented Mar 27, 2013 at 14:43
  • 1
    @BenH go to client site, say you're doing maintenance and burn a VM of their system. Bring it back and put it a 'lab management' style VM farm. Then devs will deploy to the VM, and someone else can deploy to the client once you know its perfect. Failures in dev deployment get rolled back (using the VM snapshot) until it works perfectly. The client site should never be out of date, unless someone else is also deploying there outside of your control. NEVER deploy straight to live.
    – gbjbaanb
    Commented Mar 27, 2013 at 14:46
  • Thanks Doc Brown and gbjbaanb, those are both helpful suggestions. Unfortunately, it's usually not possible to go the client site itself -- they're often offshore -- but I might be able to persuade management to pay the cost of the client FedEXing us a USB with an image of their system on it. I agree that we shouldn't deploy straight to live, it's utterly crazy. I'm trying to get this sort of dangerous breech of basic software professionalism taken seriously, but it's an uphill battle.
    – Ben H
    Commented Mar 27, 2013 at 16:27
  • I failed to get this stuff taken seriously (management preferred dealing with the flak from clients to paying the few tens of pounds for FedEX), and resigned a few months back. Somehow I care a little bit less as a contractor on twice the pay.
    – Ben H
    Commented May 1, 2014 at 20:52
2

One possible way would be to generate multiple log files (that are cycled and garbage-collected, at some size and/or frequency) and instrument your code to log things that are purely informational, warnings and errors to the log file(s) (and, ideally, have a whole bunch of debug logging that can be enabled by a flag or config setting, so you can get even more detailed logs from running in the customer environment).

These logs can then simply be zipped up by the customer and shipped to you (mail, ftp, burned to CD(s), ...)

Improve this answer
2
  • Thanks for the suggestion, clearly I wasn't clear in my original question, but I already do this. I capture the state of the application at the point of an exception and have the application email me the log. This certainly makes fixing existing bugs a lot faster, but it still has the problem of deploying code with bugs in the first place.
    – Ben H
    Commented Mar 27, 2013 at 16:24
  • @BenH Just having the state "when the exception happens" doesn't give you nearly as good data as "a whole bunch of state for 10-15 minutes before the exception happened", though. But, no, it doesn't address the root cause, that would require having a sufficiently true clone of the production environment available for testing.
    – Vatine
    Commented Mar 28, 2013 at 10:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.