Chapter 8: Performance Test Execution

Software Performance Testing Handbook
A Comprehensive Guide for Beginners

In the Internet, the competitors are only a mouse click away. Hence the web sites should provide high performing, time effective application thereby increasing the satisfaction level of the end users. Dissatisfied users would abandon the site and never come back. Bad mouthing would spoil other user’s interest towards the web site. Hence there is a high necessity to design applications with high performance to meet the expectations of the end user.

There are certain activities that need to be performed before we plan for the performance tests for a web site. By ensuring these activities, we could improve the accuracy of the performance tests and would run tests simulating the real time usage.

Quality of Service of web application


The QOS (Quality Of Service) of a web application is normally measured using the system response time and system throughput. It is about ‘How fast is the system able to respond to the customer requests?’ and ‘How much load the server can withstand by providing quick response to the customers?’ There is a high chance for losing the business when the customers are not satisfied with the QOS of the web site.

Providing QOS to the end users of the system is a big challenge faced by the application designers. The objective of the system testing is to identify the functional defects before the end user finds it. The system needs to be assured not only for its functional stability but also for its availability and responsiveness at all times of the user load. The end users will not be happy just by having a functionally stable system; the system should be very responsive and available enough at all times of user accesses.

It is very important for a web application to respond quickly with minimal processing time irrespective of number of users accessing the system. Though the application might be functionally stable for 1 user, but there could be inherent design issues or code level issues which might create show stopper issues during multi user load. Hence it is highly required to test the system performance before releasing the system to the open community.

Types of Performance Tests

There are various types of tests that could be performed on an application during the performance certification activity. The objective and importance of various types of tests are discussed below.
1. Baseline test
2. Benchmark test
3. Load test
4. Stress test
5. Soak test
6. Endurance or Longevity test

But it is not required that all application needs to undergo all of these tests during performance testing. It purely depends on the application context and its usage pattern. It is always recommended to carry out any special tests which are different from those mentioned above, if the application usage pattern demands one.
Performance Test Execution Approach

Let us recollect the basic objective of running a performance test. The Performance test is conducted to identify and certify how fast the application responds and at what point of load the application performance starts degrading. The very important expectation on the performance test activity is the ability to run more realistic tests. By injecting the required number of users load during the test, realistic end user usage pattern cannot be achieved. The server workload needs to be analyzed which is purely based on the expected accuracy level in the test results. For a performance critical application, often high accuracy is expected combined with high level of confidence about the system performance.

Normally, for performance testing, paretto’s 80-20 rule should be followed. After identifying the most frequently used business flows, user distribution among the identified business flows needs to be done carefully. Though 100% real time simulation is impossible, lot of analysis should go into this analysis to come up with more accurate realistic numbers. Usually for this analysis, web log analysis provides lot of inputs in case of applications where usage history exists. A high accuracy level can be expected in case of the availability of real time usage data. For applications, going to production for the first time, expected accuracy level should be less due to the non-availability of real time data.
It is always better to neglect those business flows which are seen or expected to be occasionally used during peak business hours. Rather during performance tests, subject the server to a constant load for 15-30 minutes approximately (also depends on application usage pattern) to get more samples of server behavior (with respect to response time and resource utilization levels) during loaded condition and then certify the application for the subjected load. A test could be called realistic only when the user load and the transaction mix are appropriate along with the associated confidence about the server performance behavior.

It is required to verify the system performance for more number of samples and then conclude the performance metrics for response time, server utilization levels, etc. The test should always have sufficient time for user ramp up, before subjecting the system to the expected load period. All the measurements about the server performance need to be taken only during the stable load period. The test should be configured in such a way that all users are actively accessing the system during the stable load period. If the peak expected server load is for example 200 requests per second, then the test should be scheduled in such a way that that running users should create a constant load of 200 requests per second atleast for 10 to 15 minutes duration. Though the peak expected user load (200 requests per second) occurs for less number of times in the real time with sufficient time interval between the spikes, the performance test should subject the system for a constant load to get more samples on the server performance. This helps in assuring the server performance behavior for a peak traffic situations. Hence the Performance Test Engineer should come up with a proper test strategy in order to test the system for realistic load and also at the same time, the test should be able to guarantee the system performance behavior for the peak load situations.

Let us discuss an example to validate the load test scenario created for the business requirements. The application needs to be validated for the 100 user load which consists of following 3 business scenarios.

Business Scenario A -60% of target load
Business Scenario B -30% of target load
Business Scenario C -10% of target load

Based on the inputs from business analysis, it was found that scenario A will be executed 6 times in an hour, scenario B will be executed 2 times in an hour and scenario C will be executed 1 time in an hour. Also, it was known that the peak server load was 200 requests per second during the peak hour.

The following load test is scheduled in order to meet the above test requirements.


This load test strategy meets the user load targets (100 users accessing the system for 40 minutes of stable load period) and the number of executions of each business scenarios (6 iterations of A scenario, 2 iterations of B scenario and 1 iteration of C scenario). During the stable load period, when all three business scenarios were running (40:00 to 50:00 mm:ss), the server load created on the server was about 200 requests per second. Hence the test seems to be realistic to the production load.

The above test scenario meets the performance test requirements of the applications and it provides the system performance figures for the realistic system usage. But what is missing in the above load test scenario is the confidence level of server being able to handle the 200 requests per second, as there was only 1 sample collected during the interval (40:00 to 50:00 mm:ss) when all three business scenarios were running. A load test should aim for assessing the server performance for realistic usage pattern and also create a high confidence by analyzing the system performance for atleast 3 to 4 samples. The confidence level increases only when more number of samples is made available and analyzed for server behavior. Also, the server resource utilization levels needs to be assured only by subjecting the system to a stable constant load. Hence it is recommended to run all the three business scenarios together for atleast 20 to 30 minutes to collect enough samples on the response time metrics and server resource utilization metrics by maintaining the load at 200 requests per second. This can be combined in the same load test or can be run as a separate test depending the application performance requirements and performance behavior of the system.

The moment we talk about realistic tests, think time should be the high priority one worth spending time for detailed analysis. In spite of having correct business flows and user distributions between the business flows, the moment think time is not realistic, the entire test goes for a toss leading to unrealistic tests. The think time decides the user request arrival pattern of an application which decides the load on the server at any point of time. Usually it is always recommended to use realistic think time between transactions and randomize it while running the test (say 80% to 120% of configured think time). This randomization helps is simulating the think time of inexperienced user of the web site and the frequent user of the web site.