Software Performance Testing Handbook
A Comprehensive Guide for Beginners
A Comprehensive Guide for Beginners
The Performance Test Reporting approach is totally different from the functional test reporting way. In the Performance Testing world, it is not easy to provide the pass/fail criteria as in the case of functional test report. The performance test report development is not as easy as everyone thinks because the test report should contain the symptoms and observations of the performance bottlenecks along with the recommendations. More than that, the information should be abstracted based on the audience of the test report.
Why is it Critical
The Performance Test Report is the document or presentation which contains the details of the performance test outcome. In case of business critical systems, the Performance Test Report would have a direct impact on the business decisions regarding the production release of the application. In some cases, the decisions related to infrastructure upgrades depend upon the test recommendations provided in the Performance Test Report. In case of products, few sections of the Test Report can also shared with all the clients as the performance benchmark. Though we should accept the fact the test report contents purely depends on the test objective, there are certain basic fundamental things which needs to be followed to develop a good test report.
How to report the results of performance testing
The foremost information to be provided in the Performance Test Report should be the overall objective of the performance testing. It should also provide the overview of the application and the business scenarios identified for testing and the user distribution model used for the testing. Though these details are provided in the test plan document, it is a good practice to include these information in the test report.
The main objective of the Performance Testing is to identify the server capacity (Throughput) to handle user load. The business might also be interested in the response time identification for the critical transactions. The Performance Testing activity provides the insight of SPEED, SCALABILITY and STABILITY of the application. The Performance Test Engineer should have this in mind while writing the test report.
The important information that should be available on the Performance Test Report includes the response time details in seconds, server throughput details in terms of transactions per second or Bytes per second , application errors or other errors encountered during the test and an indication about the hardware resource utilization levels.
Mere copying of few graphs from the performance testing tool doesn’t meet the objective of test report creation. The Performance Test Engineer should explain the test results through plain sentences in business English about what is the inference made from each of the graphs provided in the test report. In case of specific bottlenecks, symptoms needs to be explained as all the readers of the test report may not understand from looking at the graph. Each of the graphs should have its purpose detailed along with the clear details of x-axis and y-axis.
The Performance Test Engineer should ensure that right statistics are used and the report is customized to the intended audience. Also the performance test report is expected to have more visuals rather than providing the test results in text format.
Components of a good Test Report
A Performance test report will have different kind of readers like business people (management level people), technical members (architects, designers) and infra members (server admin and DBA). A good test report should meet the expectations of different kinds of readers and should make them understand the test results by adopting right level of abstraction.
It is always a best practice to have a separate section in the test report for providing the test results summary with a high level abstraction of information where technical details needs to be presented at high level but enough information on the test conclusions should be available as major business decisions are made based on this summary.
At a high-level, a performance test report should have the following information readily available.
• Performance Test objectives / goals (Service Level Agreements)
• Test environment and the server configurations
• Tests conducted (along with the deviations from the plan)
• Performance Test Result summary
• System Performance details with enough visuals
• System Performance Bottlenecks details with enough visuals
• Performance test conclusion and recommendations
Why is it Critical
The Performance Test Report is the document or presentation which contains the details of the performance test outcome. In case of business critical systems, the Performance Test Report would have a direct impact on the business decisions regarding the production release of the application. In some cases, the decisions related to infrastructure upgrades depend upon the test recommendations provided in the Performance Test Report. In case of products, few sections of the Test Report can also shared with all the clients as the performance benchmark. Though we should accept the fact the test report contents purely depends on the test objective, there are certain basic fundamental things which needs to be followed to develop a good test report.
How to report the results of performance testing
The foremost information to be provided in the Performance Test Report should be the overall objective of the performance testing. It should also provide the overview of the application and the business scenarios identified for testing and the user distribution model used for the testing. Though these details are provided in the test plan document, it is a good practice to include these information in the test report.
The main objective of the Performance Testing is to identify the server capacity (Throughput) to handle user load. The business might also be interested in the response time identification for the critical transactions. The Performance Testing activity provides the insight of SPEED, SCALABILITY and STABILITY of the application. The Performance Test Engineer should have this in mind while writing the test report.
The important information that should be available on the Performance Test Report includes the response time details in seconds, server throughput details in terms of transactions per second or Bytes per second , application errors or other errors encountered during the test and an indication about the hardware resource utilization levels.
Mere copying of few graphs from the performance testing tool doesn’t meet the objective of test report creation. The Performance Test Engineer should explain the test results through plain sentences in business English about what is the inference made from each of the graphs provided in the test report. In case of specific bottlenecks, symptoms needs to be explained as all the readers of the test report may not understand from looking at the graph. Each of the graphs should have its purpose detailed along with the clear details of x-axis and y-axis.
The Performance Test Engineer should ensure that right statistics are used and the report is customized to the intended audience. Also the performance test report is expected to have more visuals rather than providing the test results in text format.
Components of a good Test Report
A Performance test report will have different kind of readers like business people (management level people), technical members (architects, designers) and infra members (server admin and DBA). A good test report should meet the expectations of different kinds of readers and should make them understand the test results by adopting right level of abstraction.
It is always a best practice to have a separate section in the test report for providing the test results summary with a high level abstraction of information where technical details needs to be presented at high level but enough information on the test conclusions should be available as major business decisions are made based on this summary.
At a high-level, a performance test report should have the following information readily available.
• Performance Test objectives / goals (Service Level Agreements)
• Test environment and the server configurations
• Tests conducted (along with the deviations from the plan)
• Performance Test Result summary
• System Performance details with enough visuals
• System Performance Bottlenecks details with enough visuals
• Performance test conclusion and recommendations
No comments:
Post a Comment