Do think about Think Time

I got a chance to interact with lot of performance testers in various organizations , not only in India, but also in abroad & found that most of us are interested in spending time for setting realistic performance test goals & identifying the frequently used business scenarios (business flows) , but very few analyze the think time of various business flows.

It becomes very important that the average think time of the end users of the application needs to be analyzed rather than estimating specific time limit by counting the total fields to be filled by the end user. For the same application(lets take Yahoo new account creation), a new user might take 4-5 minutes or more to fill the required details to create a new account. But at the same time, a frequent old user, might take 1-2 minutes to create a new account. These think times (4-5 minutes for new user & 1-2 minutes for old user) might impact the server load.

Lets assume 100 new users are accessing the yahoo application for the time interval 13.00 to 14.00 hours & 100 old users are accessing the application for the time interval 14.00 to 15.00 hours continuously. Though there are 100 users (100 active sessions created) load on the server during both 13.00 to 14.00 hours and during 14.00 to 15.00 hours, the actual server load(requests) is going to be different during these 2 hours.

When you have a close look at the server load in requests/sec (Hits/sec) , you will find that the server load during 13.00 to 14.00 hours might be lesser than 14.00 to 15.00 hours, because of high think time used by the new users. If you look at the quantified numbers, you will be shocked to know that during the first hour, the average load could be apprx 12 hits/sec whereas during the second hour, the server load could be around 25 hits/sec.

Having performance test goals in mere user load alone doesnt make the test more realistic. Detailed analysis on the end user's think time makes it more realistic.

Response Time Vs Queue Length Vs Server Utilization

1. Is the system Response time directly / inversely proportional to the server utilization ?
2. Will there be queues(for CPU/Memory/Disk) if Utilization is less than 100% ?

Actually speaking, there is no correlation between system response time and the server utilization. The Response time increase might happen even when the server is less utililised. (To illustrate this, look at the ATM example available in the PEA site - http://www.pea-online.com/resources.htm).

Most of us dont understand this relationship. Increase in response time (during high load) is caused only because of queueing of the requests. The user request arrival pattern contributes to the high response time. Due to the adhoc user request arrival pattern, long queues are formed in various service centers which leads to high response time.

Response Time - What are the various components of Response time?

Waiting time (Queuing time) + Processing time = Response Time.

For a server, the processing time is always the same irrespective of load. For example, the server might take 2 seconds of processing during 1 user load and during 1000 users load for a transaction. But the waiting time would be high during 1000 users load, which drastically increases the overall response time. So, response time increases, if the waiting time increases which is caused by long queue.

Also, dont think that there will be queues only if Utilization is more than 100%. There can be queues formed even when the Utilization is less. So, everything boils down to knowing the user arrival pattern of the system. This is going to determine, when the system is going to be loaded and break.

Hence, knowing the user arrival pattern of a system is very important rather than setting mere goals to perform load/stress test. Analysing the user arrival pattern of an application would help in setting realistic goals which becomes more of organization logistics issue and most organizations bypass and pay for it later.

Tips to define the Performance Test Strategy

Best practices in defining performance test strategy

Most of the application would need Load tests, Stress tests and Stability(Endurance) tests to be planned before moving it to production. For data voluminous applications, there might be more emphasis on Volume testing and some applications like Auction site might emphasis on conducting spike tests with sudden spikes. For all other applications, its common that atleast 3 cycles of load tests followed by stress test and endurance test run (atleast for 12 hours) needs to be planned. (This is on a general case, but based on the application context, definetly there will be changes done to this approach).

For any application, irrepective of time availability , its always recommended to start with Baseline test. Baseline test is nothing but one user test which is run to identify the correctness of the test scripts and also it helps is checking whether the application meets the SLAs (Service Level Aggrements) for 1 user load. This values can be used a benchmark for newer versions to compare ths performance improvement.

Next comes the Benchmark test for atleast 15-20% of the target load. It helps in identifying the correctness of the test scripts and tests the readiness of the system before running target load tests.

Its a very good practice to always start with Baseline & Benchmrk tests on any application before conducting the scheduled load tests. I would say, infact its scott barber's way ofcourse.

When it comes to the scheduled tests - Load Tests, always plan to run atleast 3 roundsof load test. Irrespective of doing a slow ramp , its advised to have 3 individual load tests for 50%, 75% and 100% target load. (The load level should be defined based on the system performance rather than just 50%, 75% & 100% target load levels).

Test Scenario - Have a slow ramp up followed by a stable period for atleast an hour and ramp down. During this stable period, the target user load needs to perform various operations on the system with realistic think times. All the metrics measured should correspond only to the stable period and not during ramp up/ramp down period. Dont conclude any transaction response time just based on 1 or 2 iterations. The server should be monitored atleast for a minimum of 5 iterations (at the same load level), before concluding the response time metrics, because there could be some reason for high /less response time at any single point of time. Thats the reason watch the server performance for atleast 5 iterations at the same load level(during stable load period) and use the 90th percentile response time to report the response time metrics.

Test Metrics - Look for 90th percentile response time and standard deviation, ofcourse along with hits/sec and other server resource monitors. More the deviation(more than 1), more bursty is the graph and hence its recommended to rereun the test.

Load Tests should be always followed by Stress Tests - Based on the Load test results, slowly increase the server load step by step and find out the server break point. For this test, realistic think time settings and cache settings are not required as the objective of this test to know the server breakpoint and how it fails.

Endurance (Stability) Tests needs to be run atleast for 10-12 hours in order to identify the memory bottlenecks. This need not be run for peak load, but it can be run for average load levels.

Does Maths play a vital role in Performance Testing

Most of you must be puzzled to know Maths (Statistics) plays a vital role in Performance Testing. Yes, we must accept the fact that performance tester should use mathematical knowledge to conduct effective performance testing activity.


Slowly , (i mean it took a long time for us to understand and appreciate that mathematical knowledge is required for Performance testing) we started appreciating the fact that - a new strategical move is there in Performance testing which is nothing but the use of more & more analytical approaches to model the system more realistically and to predict the system performance for varying higher loads.


I would like to share this paper - A strategical move towards Analytical modelling with you. This paper will help you to appreciate the importance and relevance of analytical approaches for effective performance engineering activities. I prepared this paper for STEPIN 2008.

Look for the entire article here...

Capacity Planning - An ITIL Way

I prepared a paper for QAI - ITSM conference. Though it was not selected for the conference, just thought of sharing that paper with you.

This paper will provide you the glimpse of what is ITIL , what is ITSM , What is Capacity Planning, who to assure 100% system availability through effective capacity planning and how to do effective Capacity Planning.

Look for the entire article here...

Hope it helps!!!! Happy learning