Chapter 3: Deriving Performance Test Goals

Software Performance Testing Handbook
A Comprehensive Guide for Beginners

Why to set the Performance Test Goals

It is always required to decide what we are trying to achieve and then take quality steps in an organized way to achieve it. The performance test goals are important to check whether the system meets its expectations. In order to claim that the system performance is good or bad, one need performance test goals to be in place to justify the claim quantitatively.

The Performance Test Engineer needs to certify the application for performance based on the goals set by the stakeholders / derived with the help of stake holder’s inputs.

Know your customer

How to get the performance requirements from non-technical customers is a valuable question that everyone has. In most cases, customer expectations are either incompletely defined or more conceptually defined. Customers are not expected to be the performance test experts. (If so, then we are jobless!). We need to better understand their expectations by asking right set of questions and translating their conceptual requirements into more quantitative goals. The performance tester needs to play the role of a modem translating between the novice user language and performance testing terminologies. It is the performance tester’s responsibility to get the essence from the customer.

Talk in Customer’s Language: One needs to quickly understand the customer and should start talking in their language. Make a note that you don’t simply ask questions like what is the Hits/sec goal, response times expected out of various transactions and make the customer to feel uncomfortable. In one of my projects, when I asked the above set of questions, I got an unexpected answer from them, ‘Hey, in fact we also wanted to know answers for these questions. That’s why we have planned for the Performance Testing’. Though their answer surprised me; then I slowly understood that I am not talking in their language.

Look for Customer’s Expectation: Some customers may be clear in their objective; some may have conceptual expectations on the performance test results. Try to understand their language and thereby their expectation. Don’t talk using performance testing jargon and make the customer feel that they are ignorant, which is not the case. Don’t expect that your customer will give you the performance goals in a single shot. Rather , start the discussion with an overview of important terminologies like what response time actually means, what is a Hit , what is a transaction , what is a performance goal , why is it required before starting the performance tests, etc so that the customer feels comfortable to explain what he/she wants. Try to educate the customer and explain with an example to understand what they are expecting from the system.

Gain Customer’s Confidence: Try to build a good rapport with your customer. Try to make them to realize that you care for them. Try to briefly explain about how you have solved the issues in some other projects in your past. Try to make them feel that you have achieved great things in your past projects. That’s the first impression you get from them. Once you get that, you will have the full liberty to state your expectations from them in order to provide successful results. Don’t scare the customer talking too much technical rather make them to understand the results in a very polite way. More than results, your way of expressing the result and your communication style impresses the customer.

Performance Test Estimation

Performance Testing forms an investigation activity and hence it is very difficult to predict the efforts required for performance testing. But by experience I have come across various factors which impact the effort requirement for all the test activities during performance testing. As a best practice, every performance test project should get started with the project kick off meeting where the high level performance test objectives and application overview is discussed. The discussion inputs need to be considered for the effort estimation plan. The effort estimation needs to be approved by the stakeholders before continuing with the performance test plan development activity.

The performance test effort requirement depends on the performance test scope of the project. The effort required for each life cycle phase of performance testing depends on the project category. The broad classification of the performance test projects are as follows:

• Verification / ReTest: Mostly this category projects will be simple in nature and it requires a recheck on the application performance due to minor changes in the application / infrastructure.
• Application Benchmarking: Mostly this category projects will be of simple to medium complexity and it requires certification of the application for the supported load levels without any detailed bottleneck analysis or tuning activity. In most of the cases, the application is tested for the first time.
• Regression Test: Mostly this category projects will be of medium to complex category and it requires redoing of the entire performance testing for the application which was performance certified earlier due to major functional changes or fixes.
• Performance Testing: Mostly this category project will be of medium to complex category projects and it requires complete effort for all the test phases and it includes bottleneck analysis and tuning based on the project requirements. In most of the cases, the application is tested for the first time.