How to run effective performance tests

To plan for an effective performance test, follow the below four important steps

1.The first step is to understand the web site’s actual usage pattern. Mostly this information can be collected from the web server log files which are considered as the ‘gold mines’ to know every possible detail about the server load. The ‘log analyzers’, tools which helps to analyze the web log files and provide the analysis report in terms of number of user sessions, number of unique users, number of errors, peak day traffic, peak hour traffic, user request arrival pattern, frequently accessed pages, user navigation pattern, etc. From this analysis, the actual usage of the server can be identified.

2.The second step is to design the performance test approach. Based on the business forecast of the user load increase, develop service level agreement (SLAs) that has to be met by the application in order to handle the user load in the target environment. The analysis done in the first step helps to set quantitative SLA for the application. Design the test cases that needs to be tested along with the parameters like ramp up strategy, test duration, think time settings, test scenarios to be tested, different load conditions & types of tests along with the pass/fail criteria. Baseline the test approach & use this as the reference for the rest of the activities.

3.The third step is to identify the performance test tool to help us in simulating the user load on the server. Look for performance test tools available in the market (Licensed or freeware tools) based on the budget constraints & technologies supported by the test tool. Using the test tool of choice, develop the test script which simulates the actual user access pattern. Have enough test data available in the test script in order to simulate different users using different data available in the script.

4.The fourth step is to look for the readiness of the test environment. The test environment includes the application’s server infrastructure & also the load test environment. Enough care should be taken to ensure that no other users access the application during the test run & the servers are isolated from any other usage. In the case of load test environment, ensure that load generators are in place to generate the required load, the availability of test tool license to support the load to be simulated, test scripts are developed and ready with the required test data & think time, server monitors are configured with the required performance counters & the server admins are informed for their assistance in case of server failures.

Choose right set of test scenarios to schedule the performance test

Like functional testing, performance testing cannot be done for all scenarios. Identify the frequently used scenarios based on discussions with the business analyst. Sometimes, there will be couple of scenarios which have some suspections about its performance. We have to include those kind of scenarios too. It boils down to Pareto analysis – the system performance will be improved if 20% of the scenarios which are most frequently accessed are tuned. Tuning the remaining 80% of scenarios will not bring considerable performance improvement. Use Scott barber’s UCML or CBMG modeling to identify the critical scenarios.

During the performance test, don’t forget that you need to create the actual real time system usage model. (i.e.) The test should be very realistic in such a way that it models the real time traffic & more important is it should model the transaction mix of the real time workload. If you are scheduling a test for 100 users, then make sure that the 100 users are distributed across various functional flows of the application (Say in a banking application during the peak hours the users are distributed across various functionalities like, 60% of the users depositing the amount, 20% of the users transferring the amount, 10% of the users checking for the account balance & 10% of the users accessing the mini-statement). Unless this kind of mixed mode scenario is adopted for the testing, we might not end up simulating the real time user load.

There are certain situations wherein we might need to schedule the performance test with a single functional flow. One situation is when the performance testing is scheduled for each of the functionally stable build. Other situation would be when we identify some performance issues during the mixed mode scenario, then in order to isolate whether the performance issues are from a specific functional flow, we might need to run single functional flow test.

Observe the differences quoted above. As long as your objective is to test the system performance for a specific load, make sure the load is very realistic to the real time user access.

Metrics to be collected during any Performance Tests

Irrespective of the kind of performance test (load test/stress test/volume test/spike test etc), there are certain metrics which needs to be collected from the performance test tool to analyze the system performance.

End of the day , we want to know whether the system is able to handle the load meeting the response time level SLA. To achieve that we should identify the following 2 things.

1. Whether my software pipe is able to effectively utilize the available hardware pipe.

2. Whether my hardware pipe is big enough to be used by the software pipe.

If the software pipe is not able to utilize the hardware pipe effectively, then no wonder that even if you upgrade the server hardware, there will no improvement in the system performance.

Metrics to be collected

1.As a first step always look for the average & 90th percentile response time of all the transactions. Just neglect if there is other transactions which are inserted by the tool itself. (In case of Load Runner, the tool provides the transaction response time for vuser_init, vuser_end & action transactions depending upon the run time settings).
2.Next look for the load generated from the test tool. Hits/sec is the metric which provides the information on how many requests are fired to the server per second. Look for the average hits/sec & if the test is run using a specific ramp up rate, then measure the hits/sec value during the stress period of the test(when all users where running).
3.Next look for the Throughput, the metric which portrays the load handled by the server. The total transactions passed per second metric helps to identify the how capable is the server to handle the transactions. Also look for throughput in terms of bytes/sec metric which again portrays the server capability to respond to the client requests.
4.Next look for the failures occurred during the test. Observe when the failures have occurred, what is the type of failures (network failure, HTTP 500 errors, HTTP 404 errors, proxy server failures, etc) & is the failures specific for a transaction.
5.Next, look for the basic performance counters of web/application/DB server resources like %CPU, processor queue length, Memory pages/sec, Page faults/sec, % Disk utilization.

These five are the basic performance metrics which are mandatory for any performance tester to analyze the performance of the system. Going forward, in order to isolate the performance bottlenecks, we might end up looking at various performance metrics depending upon type of the performance issue.

Know the importance of Think time

In most of the performance testing approach, the ‘Think Time’ metric is taken for granted. We should understand the importance of think time before deciding the test strategy.

What is a Think Time? Think time is the time taken by the users to think or to navigate
to different pages in the application.

Depending upon the application context, the think time would vary. Its not advisable to have the default think time for all applications under testing. For the user registration page containing 5 mandatory fields, the use might take 15-20 seconds on an average. Notice the words “15-20 seconds on an average”. You can’t say that all the users would take exactly a specific think time. It would definitely vary based on each of the users. So, always make a point that whenever you run the load test, apply a randomization factor on the configured think time. Say if the think time in the script is 10 seconds between each of the transactions, configure the load test tool to use a randomized think time of 80% to 120% of the value available in the script.

As we are interested to measure the response time of the transaction (user defined actions), care should be taken to place the think time outside the transaction points. (i.e.) the think time should not be placed between the start & end of the transaction.

Most of the testers doing the performance testing don’t understand that by changing the think time, we can simulate the load of large number of users though we run the test for less number of users. To illustrate, consider the following example. Five tests were conducted with 5 virtual users simultaneously accessing the application. In each of the test, the think time is the only metric which was changed. Observer, the Hits/sec generated from the client side & Throughput (the number of transactions passed /sec).

Though the number of users hitting the server is same, the think time creates a huge impact on the server load. We need to understand the fact that decreasing the think time will increase the server load & vice versa. Always try to model the real time scenario & identify the performance figures. Never forget that we can play around with the server load just by changing the think time.

Also, dont forget to use Little's law, N = X . (R+Z) to check whats the actual load on the system for the test which you are running.

Relationship between Metrics

Things to note : Queuing Theory basics

1. The utilization of a service center is dependent on the Arrival rate.
2. The Service Demand remains constant irrespective of the load (for Load Independent devices).
3. Throughput increases with the Utilization.
4. Queues will be formed even when the utilization is less than 100% & its also possible that there can be no queues during 100% utilization too. Hence there is no relationship between the Utilization & Queue length.
5.The response time does not depend on the Utilization. It depends on the Queue length, hence the arrival pattern.
6. The maximum CPU utilization that can be achieved can be calculated using the formula Queue length / Queue Length + 1.
(i.e) For a four CPU m/c maximum utilization that can be achieved = 4/5 = 80%