A Paradigm Shift in Performance Testing

Scott Barber shares his experience in WOPR6 (sixth meeting of the Workshop on Performance and Reliability), where he happened to meet couple of great people who are known for their work in the Performance Testing field in his August 2007 column. (http://www.perftestplus.com/resources/023PeakPerf.pdf)

He happened to meet lot of great experts including Neill McCarthy, a top performance tester from U.K,Antony Marcano, a performance test consultant in U.K and manager of Testing Reflections.com,Goranka Bjedov senior Software Engineer at Google,Craig Rapin, a performance test manager of world's largest financial services,etc where he got lot of insights about Performance testing and how its gaining maturity day by day.

Am sure that if you read his article on 'Performance Testing Moves toward Maturity' , you will really feel somewhat great to think that you are also in the great research area into which there are already 10-15 pioneers who are doing great things.

Am sure that you will feel motivated.... Now the question is how are you going to withstand with this maturity.. Yes its high time for us to act immediately and to show the maturity in our way of working too :)

Directions for fellow Performance Testers

Before certifying the application performance,
are you satisfied with your own performance? are you sure that you are conducting effective performance test? Have you ever thought how to ensure the your performance test strategy and are you sure that your performance test report is bug free?

The following tips would help you to get more maturity and to do your job better

Yes..You need to be comfortable in the terminologies used in the performance engineering field and need to learn one or two popular and effective performance test tools and develop bottleneck analysis skills. That's the basic skills required for being a performance tester. Then you are matured enough to look back at your performance and look for how effective you are in your activities.

That's where, you need to start looking at Analytical modeling and simulation modeling approaches to move to your next level of expertise.

Foremost thing is that, understand the operational laws and start applying those laws in your day to day test activities.

Secondly start learning about Workload Modeling concepts and Statistical distribution fitting and apply those learning's to define performance testing goals/SLAs.

Thirdly start learning about Demand planning and Performance prediction techniques. Learn Mean value analysis and Erlang's method.

Fourth step is start learning about Capacity Planning.

This is the second level of maturity you need to gain. The next advanced third level is moving towards simulation modeling and advanced capacity planning.

Hope this helps to set your carrer path and plan to work towards it.

What is Capacity Planning in a bird's eye view?

Though the awareness about the Performance Engineering is slowly increasing, still people are thinking that Capacity planning as an alternative to performance testing or sometimes planning for the server capacities before baselining the software performance.

Capacity Planning helps in identifying the specific point of time in future through prediction when the system saturation is going to occur and provides the most cost-effective way of delaying system saturation as long as possible.

Capacity Planners help in identifying the system's saturation point. When and why the saturation would occur and which are the best cost-effective alternatives for avoiding the saturation.

Before looking at the underlying server hardware, we need to first look at the software performance and perform tuning activities to get better performance. Then we need to think about the server capacities.End of the day, its the server's workload pattern which consumes the system resources. Identify the system workload and analyze the system resource usage for the typical workload expected on the server. Based on this, arrive at the capacity baseline on the current server hardware.

Based on the business forecasts, do a prediction on how much increase would be expected in the workload due to the business forecast and identify the required server resources to support the demand. Thus Capacity Forecast is done for the forecasted load expected by the business.

Things to note before you attend interview for Performance Testing role

Based on my experience as a Interviewer, i would like to share few things which the interviewee must follow/know before attending the interviews for Performance tester role.Couple of things which i felt its really missing in many interview candidates is what i have compiled and provided below. I feel that the below skills are mandatory for any performance tester for 1-2 years of experience.

1. Understand that LoadRunner/Silk Performer knowledge is not everything. Its just a tool which enables us to do better performance testing.

2. Be good in Performance testing terminologies - Types of tests and its objectives,various Performance metrics,monitors,etc. The differences/changes required in scheduling each type of tests should be known.Irrespective of whether you have the responsibility for deciding the test strategy , always know when and why to conduct each of the tests.

3. First, be clear on the objective of conducting performance test for any application. I mean set your own objective that you (along with the team) need to make the system to meet its performance objective. Plan realistic tests accordingly as per the business understanding and try your best to identify various load situations that make the system to fail and work as a team to resolve those bottlenecks. Its not enough to conduct some load tests and say that application doesn't meets it Throughput SLA or response time SLA.How you identify and isolate the bottleneck is more important. What and How you observe and interpret during the test is very important.

4. As a Performance tester, you should know the basic relationship between Hits , Response time , Throughput, etc. This forms the basic for the bottleneck analysis. Increase in response time is not a bottleneck you have identified, its the symptom for some bottleneck. Its a must to identify the hidden bottleneck rather than thinking that's its out of scope of your testing. Its your responsibility to run various isolation tests to gather observations about system performance and provide your suggestions to the DB admins and system admins. Dont forget that its your responsibility to get those people involved to help you out in isolating the issues. I agree that, its a team effort but the sole leader here is the Performance tester who need to coordinate all the other members to get the bottleneck identified. Always think that they are going to help you rather than thinking that you are helping them. That thought makes lot of difference in the ownership you have and the responsibilities you hold.

5. If you are an expert in a specific performance test tool, rank your competency appropriately. Explain what you know and what you don't know. State the reasons for what you don't know and why couldn't use\know that. All the organizations will not look for tool expert who knows everything about a specific tool. Having the tool knowledge is important, but that's not everything. So, be frank enough to tell what you know and what you don't know. How much you are conceptually good is what matters in most of the cases.

6. Atleast in Performance Testing/Engineering field, i definitely feel that a good performance engineer should have referred lot of books/articles/white papers if he/she is really looking for a way to conduct more effective performance tests. Talk about how it made a difference in you/your project.

7. On the whole, the performance tester should include the tool knowledge(1 or 2 tools),knowledge and experience for doing performance requirements analysis to set performance SLAs, Test environment creation details , Bottleneck analysis skills, Basic knowledge about the technology used in the applications and basic knowledge about the OS , Server software & hardware details. (I have just stated the core required skills).

Hope it helps you to relook at your skills and identify the gaps before you attend your next interview.