Tips to define the Performance Test Strategy

Best practices in defining performance test strategy

Most of the application would need Load tests, Stress tests and Stability(Endurance) tests to be planned before moving it to production. For data voluminous applications, there might be more emphasis on Volume testing and some applications like Auction site might emphasis on conducting spike tests with sudden spikes. For all other applications, its common that atleast 3 cycles of load tests followed by stress test and endurance test run (atleast for 12 hours) needs to be planned. (This is on a general case, but based on the application context, definetly there will be changes done to this approach).

For any application, irrepective of time availability , its always recommended to start with Baseline test. Baseline test is nothing but one user test which is run to identify the correctness of the test scripts and also it helps is checking whether the application meets the SLAs (Service Level Aggrements) for 1 user load. This values can be used a benchmark for newer versions to compare ths performance improvement.

Next comes the Benchmark test for atleast 15-20% of the target load. It helps in identifying the correctness of the test scripts and tests the readiness of the system before running target load tests.

Its a very good practice to always start with Baseline & Benchmrk tests on any application before conducting the scheduled load tests. I would say, infact its scott barber's way ofcourse.

When it comes to the scheduled tests - Load Tests, always plan to run atleast 3 roundsof load test. Irrespective of doing a slow ramp , its advised to have 3 individual load tests for 50%, 75% and 100% target load. (The load level should be defined based on the system performance rather than just 50%, 75% & 100% target load levels).

Test Scenario - Have a slow ramp up followed by a stable period for atleast an hour and ramp down. During this stable period, the target user load needs to perform various operations on the system with realistic think times. All the metrics measured should correspond only to the stable period and not during ramp up/ramp down period. Dont conclude any transaction response time just based on 1 or 2 iterations. The server should be monitored atleast for a minimum of 5 iterations (at the same load level), before concluding the response time metrics, because there could be some reason for high /less response time at any single point of time. Thats the reason watch the server performance for atleast 5 iterations at the same load level(during stable load period) and use the 90th percentile response time to report the response time metrics.

Test Metrics - Look for 90th percentile response time and standard deviation, ofcourse along with hits/sec and other server resource monitors. More the deviation(more than 1), more bursty is the graph and hence its recommended to rereun the test.

Load Tests should be always followed by Stress Tests - Based on the Load test results, slowly increase the server load step by step and find out the server break point. For this test, realistic think time settings and cache settings are not required as the objective of this test to know the server breakpoint and how it fails.

Endurance (Stability) Tests needs to be run atleast for 10-12 hours in order to identify the memory bottlenecks. This need not be run for peak load, but it can be run for average load levels.

Does Maths play a vital role in Performance Testing

Most of you must be puzzled to know Maths (Statistics) plays a vital role in Performance Testing. Yes, we must accept the fact that performance tester should use mathematical knowledge to conduct effective performance testing activity.


Slowly , (i mean it took a long time for us to understand and appreciate that mathematical knowledge is required for Performance testing) we started appreciating the fact that - a new strategical move is there in Performance testing which is nothing but the use of more & more analytical approaches to model the system more realistically and to predict the system performance for varying higher loads.


I would like to share this paper - A strategical move towards Analytical modelling with you. This paper will help you to appreciate the importance and relevance of analytical approaches for effective performance engineering activities. I prepared this paper for STEPIN 2008.

Look for the entire article here...

Capacity Planning - An ITIL Way

I prepared a paper for QAI - ITSM conference. Though it was not selected for the conference, just thought of sharing that paper with you.

This paper will provide you the glimpse of what is ITIL , what is ITSM , What is Capacity Planning, who to assure 100% system availability through effective capacity planning and how to do effective Capacity Planning.

Look for the entire article here...

Hope it helps!!!! Happy learning

A Paradigm Shift in Performance Testing

Scott Barber shares his experience in WOPR6 (sixth meeting of the Workshop on Performance and Reliability), where he happened to meet couple of great people who are known for their work in the Performance Testing field in his August 2007 column. (http://www.perftestplus.com/resources/023PeakPerf.pdf)

He happened to meet lot of great experts including Neill McCarthy, a top performance tester from U.K,Antony Marcano, a performance test consultant in U.K and manager of Testing Reflections.com,Goranka Bjedov senior Software Engineer at Google,Craig Rapin, a performance test manager of world's largest financial services,etc where he got lot of insights about Performance testing and how its gaining maturity day by day.

Am sure that if you read his article on 'Performance Testing Moves toward Maturity' , you will really feel somewhat great to think that you are also in the great research area into which there are already 10-15 pioneers who are doing great things.

Am sure that you will feel motivated.... Now the question is how are you going to withstand with this maturity.. Yes its high time for us to act immediately and to show the maturity in our way of working too :)

Directions for fellow Performance Testers

Before certifying the application performance,
are you satisfied with your own performance? are you sure that you are conducting effective performance test? Have you ever thought how to ensure the your performance test strategy and are you sure that your performance test report is bug free?

The following tips would help you to get more maturity and to do your job better

Yes..You need to be comfortable in the terminologies used in the performance engineering field and need to learn one or two popular and effective performance test tools and develop bottleneck analysis skills. That's the basic skills required for being a performance tester. Then you are matured enough to look back at your performance and look for how effective you are in your activities.

That's where, you need to start looking at Analytical modeling and simulation modeling approaches to move to your next level of expertise.

Foremost thing is that, understand the operational laws and start applying those laws in your day to day test activities.

Secondly start learning about Workload Modeling concepts and Statistical distribution fitting and apply those learning's to define performance testing goals/SLAs.

Thirdly start learning about Demand planning and Performance prediction techniques. Learn Mean value analysis and Erlang's method.

Fourth step is start learning about Capacity Planning.

This is the second level of maturity you need to gain. The next advanced third level is moving towards simulation modeling and advanced capacity planning.

Hope this helps to set your carrer path and plan to work towards it.