Modern businesses require their application systems to perform in a responsive and cost-effective way. To develop such systems, performance regressions have to be identified early during the software development process, and especially before deployment. However, practice shows that load testing, as a relevant form of performance (regression) testing, is either not performed adequately or not at all. Load tests are usually not representative enough for the actual usage profile in production. This situation is caused by the fact that they require additional effort and necessary competence to create load test scripts. In addition, these scripts become obsolete very quickly due to system and usage profile changes, and their maintenance is associated with high costs. In the context of emerging continuous software development (DevOps), in which software changes are being put into operation ever faster and more frequently, this becomes even a greater problem. Furthermore, because the execution of these tests requires a longer time, it is not practical to carry out a load test for each change, e.g., to detect performance regressions. However, if regressions are detected in these aggregated changes, manual diagnosis is necessary to determine the responsible change in the software or the usage profile.
The ContinuITy research project, executed by NovaTec Consulting GmbH and the University of Stuttgart (Reliable Software Systems Group), aims to ensure automated and efficient load testing by integrating continuously recorded measurement data from production and to integrate it into the continuous software development environment. Load tests are automatically extracted from APM (application performance management) data and evolved taking into account changes in the usage profile. To specify load tests, we use a modular description language, which can be extended with additional contextual information, e.g., with test type and objectives. As a part of the automation of the software development process (continuous delivery), relevant load tests are selected to be executed in testing. Obtained results are used for the detection of regressions and their diagnosis. ContinuITy examines regressions to find whether they are caused by changes in the implementation or the usage profile and provides results to the DevOps team
diagnoseIT is sponsored by the German Federal Ministry of Education and Research under Grant No. 01IS17010.