How to include 100k+ users performance tests on a CI pipeline

Question

I'm currently working on a architectural design for the testing of a product that's estimated to have over 100k simultaneous user hits. For the product specification, I'm including continuous integration and continuous delivery practices by using cloud CI services as Travis.

Now, I want to also stress/performance test the product against a simulated 100k simultaneous requests scenario. At first glance, a [not so] quick solution would include setting up an entire testing environment (outside Travis, of course), literally replicating the production environment, and using a dedicated system to spawn 100k separate processes, hitting the application running on the testing environment, and collecting reports.

Now, this approach feels very basic and very close to a bad practice (specially because I came up with this while I wrote this post). There is probably a standard design or solution (maybe even as a service) to solve this. Also, I don't see how would this fit on a CI pipeline.

I also want to add that covering this non-functional requirement (app should handle simultaneous 100k users) looks like it's probably solved by a good infrastructure, rather than code. Probably what I'd be needing to work with are load balancers and server providers that would let me scale the resources horizontally. However, even though this is DevOps, I want to run performance tests on it.

So how can I perform performance/load test for 100k users adding it to my CI pipeline?

Answer 1

You (almost) already said the answer:

a [not so] quick solution would include setting up an entire testing environment (outside Travis, of course), literally replicating the production environment, and using a dedicated system to spawn 100k separate processes, hitting the application running on the testing environment, and collecting reports.

Yes, this is a good solution. And, even better, it will force you to automate your production set-up. There is a way to do it that is not to use a "dedicated" system, instead to spin up and tear down the entire system so that it only runs when you need it.

Use a cloud service like AWS to spin up a production-capable cluster in seconds or minutes (and also able to tear it down in seconds or minutes), and then use a cloud service like AWS to spin up hundreds or thousands of instances to test your app.

This is definitely possible and will force you and your team to adopt a whole lot of "best practices". You don't have to do this in AWS, I'm sure that Azure or Google Cloud should also work just fine.

Disclaimer: I attended AWS reInvent a few years ago and saw a talk by a company that did exactly this. They had CloudFormation scripts that could spin up an entire production cluster (in a VPC no less) in less than ten minutes, and then used thousands of spot instances to hammer on the production cluster. It ran daily in a matter of minutes and the cost was very cheap.

Answer 2

Since performance testing is almost totally dependent on your production or deployment environment such tests in your development or CI environment are almost meaningless unless that environment is a very close match to your final environment including any additional loads or constraints that your production environment has.

One example that I remember was a system that when tested on the test environment performed perfectly but on the real deployment was unusable. The test environment had a direct, very fast, connection so the large, beautiful, graphics showed up nearly instantly but most users were on 28k BAUD modems.

Your test will also likely take a long time to run which is not ideal in a CI environment.

You can, however, test that it is not constrained to less than 100k requests by either locating a long running request or having a special test mode request that does not terminate unless signalled to do so and having a single client open 100k such requests each on a new connection.

Answer 3

I believe you need to perform the main performance test separately on ad-hoc basis as it is much more than just checking response time or throughput. So I would recommend running load testing, soak testing, stress testing, etc. separately to find out any bottlenecks, memory leaks, infrastructure problems, etc. and one you find out everything and fix it and will be happy with how does it perform you could create a scaled down test for regression purposes.

1. Run main load test with 100k virtual users
2. Identify saturation/breaking points, make performance fixes, etc.
3. Run scaled down load test with 1k virtual users to get baseline performance metrics
4. Add this scaled down load test with 1k virtual users to run on each build for regression purposes so if new features or bug fixes will cause performance degradation you will be notified.

See Running a JMeter Test via Jenkins Pipeline - A Tutorial article for example configuration.

Answer 4

Performance testing should be done on hardware that is at parity with your production environment. Otherwise you can't tell whether your results will match what your true users will see.

That being said, there are some things you can do to estimate by establishing a baseline.

1. Perform a full cycle of tests on your preproduction/performance testing environment and get the system to a point where it meets your SLAs.

2. Run the same cycle in your CI environment or on a virtual machine. Be sure to use identical code. It will run at a much lower level of performance, of course.

3. Since you know the results from #1 are acceptable, and #1 and #2 are running identical code, then you can use results from #2 as a baseline.

4. Continue running performance tests in your CI environment, and raise an alert if performance ever gets worse than the baseline established in step 2.

Of course, performance doesn't scale the same way on a VM as it would on your multi-node production web farm, so we have to make a lot of assumptions to embrace these results. If it gets 10% slower in your CI environment, it may not correspond to a 10% loss in production-- it could be lower, or it could be higher. Even if it stays the same, you can't necessarily assume that performance wasn't impacted in some way, because perhaps your CI environment is differently sensitive to different variables. Nonetheless, it is better than nothing.