1

We recently clustered an application, and it came to light that because of how we're doing SSL offloading via the load balancer in production it didn't work right. I had to mimic this functionality on my local machine by SSL offloading Apache with a proxy, but it still isn't a 1-to-1 comparison. Similar issues can arise when dealing with stateful applications and sticky sessions. What would be the industry standard for testing this kind of production "black box" scenario in a local environment, especially as it relates to clustering?

6

It can be expensive and painful, but in the end you need to have a local "cluster". Trying to simulate race conditions, contention and the like are very hard on a single PC (my interpretation of "local development environment").

From past experience I would suggest:

  • Push very hard to get a production level cluster into your test/dev environment, You can scale back some to keep costs down, but the profile needs to be representative of your production environment.
  • Get periodic (weekly?) refreshes of your production DB into your dev/test environment
  • Instrument your code for high amounts of logging in this environment
  • Implement remote debugging on this cluster as best you can given your particular language and deployment stack

I have used this strategy in the past and was able to identify the underlying bug. Then I was able to develop locally and unit test locally before deploying the fix to the test cluster.

  • I was going to answer this way, but I have never worked with clusters, so I refrained. But the embedded world has similar issues. Try as you might to simulate the hardware, once you run on the real hardware there's always stuff that doesn't work as expected. I suspect the same goes for clusters and your only good option is to have a software lab setup to mimick the production environment. Maybe it can be done on the cheap with using lesser quality systems than production uses. – Dunk Nov 1 '13 at 17:33
  • 1
    Agreed, simulation can only get you so far. Some problems need to be worked on in the true environment, whether that be clustered, embedded or some other architecture... – cdkMoose Nov 1 '13 at 18:08
  • While this is the optimum solution I do wonder about the cost effectiveness. Our network operations finally received the budget for a staging instance of our load balancer, but that was an expensive decision. I'm curious if there are open source load balancers that emulate their higher load professional counterparts. I know it still isn't entirely 1-to-1, but maybe a little closer? – user9483 Nov 1 '13 at 18:15
  • You may be able to get away with cheaper replacements for some parts, but in the end, you are debugging a problem with a specific set of hardware/software and replacing any component may leave you unable to replicate. Admittedly, a tough decision to make financially, but if this is a external product or an internal critical application, you will need this level of test environment sooner or later. Cost effectiveness should take into account loss of business/functionality, if you can not properly provide support. – cdkMoose Nov 1 '13 at 18:28
  • @BrianReindel As long as you have an environment where you can hash out these issues it shouldn't be that big a deal. Staging could work as a long as they will let you get in there to test and fix these types of issues before they go public. – stonemetal Nov 1 '13 at 20:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy