3

I'm currently working on a architectural design for the testing of a product that's estimated to have over 100k simultaneous user hits. For the product specification, I'm including continuous integration and continuous delivery practices by using cloud CI services as Travis.

Now, I want to also stress/performance test the product against a simulated 100k simultaneous requests scenario. At first glance, a [not so] quick solution would include setting up an entire testing environment (outside Travis, of course), literally replicating the production environment, and using a dedicated system to spawn 100k separate processes, hitting the application running on the testing environment, and collecting reports.

Now, this approach feels very basic and very close to a bad practice (specially because I came up with this while I wrote this post). There is probably a standard design or solution (maybe even as a service) to solve this. Also, I don't see how would this fit on a CI pipeline.

I also want to add that covering this non-functional requirement (app should handle simultaneous 100k users) looks like it's probably solved by a good infrastructure, rather than code. Probably what I'd be needing to work with are load balancers and server providers that would let me scale the resources horizontally. However, even though this is DevOps, I want to run performance tests on it.

So how can I perform performance/load test for 100k users adding it to my CI pipeline?

2

You (almost) already said the answer:

a [not so] quick solution would include setting up an entire testing environment (outside Travis, of course), literally replicating the production environment, and using a dedicated system to spawn 100k separate processes, hitting the application running on the testing environment, and collecting reports.

Yes, this is a good solution. And, even better, it will force you to automate your production set-up. There is a way to do it that is not to use a "dedicated" system, instead to spin up and tear down the entire system so that it only runs when you need it.

Use a cloud service like AWS to spin up a production-capable cluster in seconds or minutes (and also able to tear it down in seconds or minutes), and then use a cloud service like AWS to spin up hundreds or thousands of instances to test your app.

This is definitely possible and will force you and your team to adopt a whole lot of "best practices". You don't have to do this in AWS, I'm sure that Azure or Google Cloud should also work just fine.

Disclaimer: I attended AWS reInvent a few years ago and saw a talk by a company that did exactly this. They had CloudFormation scripts that could spin up an entire production cluster (in a VPC no less) in less than ten minutes, and then used thousands of spot instances to hammer on the production cluster. It ran daily in a matter of minutes and the cost was very cheap.

  • Even if you manage your own hardware, scripting the deployment process so that it can be done easily and frequently is crucial. With that, replicating the production environment ought to be fairly trivial. – amon Jun 25 '17 at 10:01
  • Eddie, have you actually done this somewhere you work, or did you just hear about it at reInvent? I am curious how you know this is "definitely possibly" without knowing any of the OP's NFRs. – John Wu Jun 25 '17 at 19:55
  • @JohnWu well I'm sure it's technologically possible. If the OP's current production is on-premise or co-located in a D.C. and they have some policy that prevents their code from being moved outside of infrastructure that the company owns, then it would not be an option. But if the OP is tasked with solving the problem, then what I've described should be an option presented to management, with a set of Pros and Cons. You could build your own virtualization infrastructure and do the same within the company but the cost would likely be prohibitive. The cloud option would cost just a few dollars. – RibaldEddie Jun 25 '17 at 20:02
  • Right. Have you done this yourself? – John Wu Jun 25 '17 at 20:10
  • I'd prefer not to say, since it might be possible for people out there who know my StackOverflow handle to discern proprietary information about previous employers. But I know of at least one company that has done this for sure, and I have definitely personally built a system capable of performing this kind of automated provisioning, orchestration, and deployment. I've even open sourced the start of a system capable of doing this. It's kind of a dead project though-- haven't touched it in 4 years-ish: github.com/eep/platform – RibaldEddie Jun 25 '17 at 20:22
1

Since performance testing is almost totally dependent on your production or deployment environment such tests in your development or CI environment are almost meaningless unless that environment is a very close match to your final environment including any additional loads or constraints that your production environment has.

One example that I remember was a system that when tested on the test environment performed perfectly but on the real deployment was unusable. The test environment had a direct, very fast, connection so the large, beautiful, graphics showed up nearly instantly but most users were on 28k BAUD modems.

Your test will also likely take a long time to run which is not ideal in a CI environment.

You can, however, test that it is not constrained to less than 100k requests by either locating a long running request or having a special test mode request that does not terminate unless signalled to do so and having a single client open 100k such requests each on a new connection.

1

I believe you need to perform the main performance test separately on ad-hoc basis as it is much more than just checking response time or throughput. So I would recommend running load testing, soak testing, stress testing, etc. separately to find out any bottlenecks, memory leaks, infrastructure problems, etc. and one you find out everything and fix it and will be happy with how does it perform you could create a scaled down test for regression purposes.

  1. Run main load test with 100k virtual users
  2. Identify saturation/breaking points, make performance fixes, etc.
  3. Run scaled down load test with 1k virtual users to get baseline performance metrics
  4. Add this scaled down load test with 1k virtual users to run on each build for regression purposes so if new features or bug fixes will cause performance degradation you will be notified.

See Running a JMeter Test via Jenkins Pipeline - A Tutorial article for example configuration.

0

Performance testing should be done on hardware that is at parity with your production environment. Otherwise you can't tell whether your results will match what your true users will see.

That being said, there are some things you can do to estimate by establishing a baseline.

  1. Perform a full cycle of tests on your preproduction/performance testing environment and get the system to a point where it meets your SLAs.

  2. Run the same cycle in your CI environment or on a virtual machine. Be sure to use identical code. It will run at a much lower level of performance, of course.

  3. Since you know the results from #1 are acceptable, and #1 and #2 are running identical code, then you can use results from #2 as a baseline.

  4. Continue running performance tests in your CI environment, and raise an alert if performance ever gets worse than the baseline established in step 2.

Of course, performance doesn't scale the same way on a VM as it would on your multi-node production web farm, so we have to make a lot of assumptions to embrace these results. If it gets 10% slower in your CI environment, it may not correspond to a 10% loss in production-- it could be lower, or it could be higher. Even if it stays the same, you can't necessarily assume that performance wasn't impacted in some way, because perhaps your CI environment is differently sensitive to different variables. Nonetheless, it is better than nothing.

  • So you're going to run 100k users against your VM? How would that work? I would contend that the profile of a single VM vs a production cluster are so different, you can't reason about the differences at all. You're doing voodoo magic here, not sound analysis. – RibaldEddie Jun 24 '17 at 5:24
  • I can't say I disagree-- I thought I said all that-- but like I said, it is "better than nothing," for those of us who can't convince our boss to pony up $XM for a dedicated performance testing environment. At least it is better than waiting for someone to say "It seems slower." – John Wu Jun 24 '17 at 7:15
  • If you see my answer, it certainly doesn't cost millions of dollars. – RibaldEddie Jun 24 '17 at 13:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.