Software Engineering Stack Exchange is a question and answer site for professionals, academics, and students working within the systems development life cycle. It only takes a minute to sign up.
Sign up to join this community
Do you do any performance testing in any repeatable and automated way? Do you trigger it as a part of CI builds? What tools do you use? How do you spot changes in trends?
I'm looking for how-tos and description of working set-ups for end-to-end automated performance testing approaches.
Repeatable - yes. Automated - not fully. We have automated the generation of test data for our performance tests, and the test driver. The tests themselves are triggered by hand whenever necessary. Such as before releases, or platform changes (right now we are migrating our legacy project from Java 5 to 6, and we are about to conduct performance tests to compare the results of different JVM settings). So they are not part of our CI builds. That would in fact be technically challenging as we need to regenerate a lot of test data each time, and the process, albeit automated, is time consuming.
We use mostly tools developed in-house for the testing, tailored specifically for our app. The app is a frontend to an ancient mainframe server, which is the backbone of the whole company. The test data needs to be generated on the mainframe, so there is lots of communication overhead. Also, it can't be really randomized at the moment - we use a set of predefined templates combined with data coming from Excel sheets to create new test data. The test driver then sends requests to our server in standard HTTP, and a proprietary XML based format.
We are just getting started with load and performance testing. Right now, we're doing simple tests: Run operation X for Y iterations, possibly with some level of randomness to inputs or things that could happen during operation (e.g., close the connection occasionally and make the program recover). Then automatically verify various interesting facts - e.g., that all Y operations completed successfully in under Z seconds, no errors were logged, etc. Note that there are a lot of things to define before you can test. What is "completed"? What is "success"? How big should Z (the number of seconds) be?
As I said, we are just getting started; we made our first test last week, literally (we began work on the product a couple of months back). We actually are setting up CI at the same time, and haven't put testing and CI together yet. I don't think we will be putting load / performance tests into CI right now, but we could do it eventually for small operations by running them for some number of iterations - probably estimated to be 5 to 10 seconds of work - then verifying that the operations took the expected amount of time, and checking any other aspects of load or performance that we're uncertain about.
Currently, we're doing a nightly run that includes our end-to-end tests (which will take 3 to 4 minutes each) and our load tests.
Our system is currently pretty simple. We're creating special performance test fixtures that are just like our normal fixtures except that they run the same operations over and over, then verifying results by checking database timestamps, counting operations, or looking at logs. Later, we'll have to worry about load-testing web sites, imitating users, etc., and we're still figuring out what tools we can use to handle that problem.
