We recently clustered an application, and it came to light that because of how we're doing SSL offloading via the load balancer in production it didn't work right. I had to mimic this functionality on my local machine by SSL offloading Apache with a proxy, but it still isn't a 1-to-1 comparison. Similar issues can arise when dealing with stateful applications and sticky sessions. What would be the industry standard for testing this kind of production "black box" scenario in a local environment, especially as it relates to clustering?
It can be expensive and painful, but in the end you need to have a local "cluster". Trying to simulate race conditions, contention and the like are very hard on a single PC (my interpretation of "local development environment").
From past experience I would suggest:
- Push very hard to get a production level cluster into your test/dev environment, You can scale back some to keep costs down, but the profile needs to be representative of your production environment.
- Get periodic (weekly?) refreshes of your production DB into your dev/test environment
- Instrument your code for high amounts of logging in this environment
- Implement remote debugging on this cluster as best you can given your particular language and deployment stack
I have used this strategy in the past and was able to identify the underlying bug. Then I was able to develop locally and unit test locally before deploying the fix to the test cluster.