Chapter 1: Introduction

Software Performance Testing Handbook

A Comprehensive Guide for Beginners

Why Performance Testing is Crucial

Performance Testing is an essential activity for developing optimal software solutions. It is highly important to avoid last minute surprises that arise after deploying the application in the production environment. The failures experienced by the end users are vulnerable to the competitors.

Performance Testing helps to answer the sample of following questions:

• What is the response time of system during the expected load condition?
• How is my system behaving during unexpected load condition?
• Is my system scalable to specific user load?
• Which environment provides the best performance for my system?
• Is my system performing well after adding the new NIC card?
• Will my system handle the spike user loads? Or will it crash?
• Does my system need a dedicated database server to meet the performance goals?

The performance being the essential quality of software application, it is required to evaluate the software systems for its performance.

Performance Testing Jargons

The following are the some of the main performance testing terminologies used in day today life. A good understanding of the below jargons are critical to get the most out of this book.

Business Transaction: It refers to a sequence of request-response and the related information exchange between the client and the server in order to accomplish a business requirement. For example, an online banking customer transferring money from one account to other is a business transaction.

Test Scenario: It refers to a high critical business transaction or high traffic workflow identified for the performance testing. Normally all the business transactions cannot be tested for its performance. The Paretto analysis helps to identify those specific 20% transactions which is often exercised by the end users rather than testing the remaining 80% non-critical transactions.

Think Time: It refers to the time taken by the user for thinking or clicking on any of the web page links, buttons, etc while navigating through the web site. Think time is a very important parameter which needs to be set while scripting the scenarios using the performance testing tools. The business analysts of the application or web site management team or sometimes even the end user survey might give a realistic picture about the think time requirements of a transaction.

Virtual User: The Performance testing tool simulates the real user traffic by creating virtual users during the performance test. A virtual user is configured in the performance test tool to run the script simulating the real user behavior. Though in real time, users access the application from a unique IP address, more than one virtual user can be configured to access the application from the same IP address.

Simultaneous User load: The simultaneous users have the active session on the server at any point of time wherein each user will be executing different transactions. For example, if we say 100 simultaneous users load, then there will be 100 active sessions opened up in the server, wherein each user will be performing different set of transactions – one logging in, another viewing reports, another navigating to the next page, etc. The simultaneous user load of a web site would be always greater than the concurrent user load of a web site.

Concurrent User load: The concurrent users connect to the server and perform the same operation at any point of time. For example, if we say 100 concurrent user load, all 100 users would be logging in at the same point of time, view the reports at the same point of time, etc. For example, an online banking web site might have 10,000 – 20,000 simultaneous user load, but 1000 to 1500 concurrent user load.

Performance Requirements/Goals: It refers to the quantitative way of putting forth the criteria for claiming the system performance is good. The performance requirements of an application are often expressed in response time, hits per second, transactions per second, etc. The performance requirements of the System Under Test (SUT) need to be quantitatively expressed in the performance test plan.

Workload: It refers to the user load subjected by a web application under real time user access or during the performance test and the way the users are distributed between various transaction flows. Normally, the web server log files of a web site can be analyzed to learn about the workload of the site (if the web site is already in production). For web sites which are yet to go alive for the first time, a workload model needs to be developed based on the discussion with business analysts, application experts, etc. It is very important to know the workload of the application before conducting the performance test. Conducting the performance test for a system without proper analysis on the workload might lead to misleading results.

User Abandonment: It refers to the situation wherein the end users exit from the web site because of the slow performance of the site. The user abandonment rate varies from web site to web site. A low priority web site might experience a high user abandonment rate compared to the payroll web site. Analysis on the abandonment rate of the site needs to be done before concluding on the load on the site.

Baseline Test: It refers to the test conducted to measure the application performance for 1 virtual user load. The baseline test is often conducted to collect the metrics about the system performance for 1 user load and thereby it could be used for comparing the system performance at a high load condition. It also helps to check the validity of the test script.

Benchmark Test: It refers to the test conducted to measure the application performance for a low load condition. Normally for a benchmark test, 15 -20% of the target load can be considered. The objective of the benchmark test is to validate the correctness of the test scripts and to check the readiness of the system before subjecting it to a high load. It helps to know whether various components of the system collaborate as per the design and meet the performance test SLA for 20% of target load.

Hit: It refers to the server request for accessing a web page or a file or an image from a web server. For example, if a web page contains 5 images, then a user visit to that page creates 6 hits on the web server (5 hits for fetching each of the images and 1 hit for fetching the web page). For a web site, the performance requirement can be derived in terms of hits per unit time like ‘the system should support the load of 10 hits per second with a response time of below 5 seconds’.

Response Time: It refers to the servicing time or processing time in order to serve the request. It is measured from the time the web browser sends the request to the web server to the time when the first byte of response is received by the requesting user after server processing. The response time of a transaction say login refers to the time taken by the server to respond to the request by logging into the system and displaying the home page (it includes web server processing time + application server processing time + database server processing time + network latency).

Throughput: It refers to the amount of data (in bytes) transferred by the server in order to the serve the client requests. It is a good indicator of server performance as it refers to the amount of work done by the server. Throughput also referrers to the number of requests or transactions processed by the server at any point of time. For example, the server throughput can be expressed as 2.5Mbps or 35 Hits/sec or 8 Transactions/sec.

Page Views: It refers to the number of pages transferred by the server in order to serve the client requests. i.e. number of times the web site is completely loaded or refreshed. A page is a html page or script page or plain text documents. It is also a good indicator of server performance. If a user visits a web site and navigates to 5 pages and then logs off then the Page Views could be indicated as 5.

Performance Bottleneck: It refers to the slow spot, the effects of which are widely felt. It refers to the situation/areas which do not allow the application to perform as per its ideal specifications. For example, the response time increase for the load of 100 virtual users because of improper setting of HTTP connections parameter in the IIS server, CPU utilization reaching 95% during 100 users load are typical performance bottlenecks. A bottleneck might lead to a failure if mitigation actions are not taken.

Load Test: It refers to the test conducted on the system simulating the actual usage pattern or a representative load in order to identify the system behavior and diagnose the performance bottlenecks in the system. The load test should have the objective of checking how the system performs for the specific load, say 1000 simultaneous users. It is carried out by ramping up the user load from zero to the maximum count with the specific ramp up rate depending upon the kind of application and the user access pattern. Sometimes load testing is conducted to identify the maximum acceptable user load or the highest arrival rate the system can handle.

Stress Test: It refers to the test conducted by subjecting the system to an unreasonable load to identify the system breakpoint. It is generally considered as a negative testing as the system is subjected to an unrealistic load. It is usually conducted to know the importance of potential harm created during unrealistic load and to act on it proactively. It also sometime helps the web site owners to inform the user base about the scalability limit of the application.

Spike Test: It refers to test conducted by subjecting the system to a short burst of concurrent load to the system. This test might be essential while conducting performance tests on an auction site wherein a sudden load is expected.

Volume Test: It refers to the tests designed to measure the throughput of the system more in a batch processing, messaging kind of environment. The objective of this test is to identify the processing capacity. Also, in database server perspective , a volume test can be conducted to test the server performance by subjecting it to huge data (say , 75,000 rows of employee information in EMPLOYEE table) keeping in mind the data available by end of 2 years of web site usage.

Stability / Longevity / Endurance Test: It refers to the tests conducted to check whether the system is able to withstand the load continuously for a longer duration say 48 hours, etc. The system is subjected to reasonable representative load and tested for its continued performance. These tests are helpful in identifying memory problems (memory leaks, buffer overflow, etc) which arise after continuous usage of the application for a longer duration.

Bottleneck Isolation Test: It refers to the specific tests conducted on the system or a specific component of a system to isolate or confirm the performance problem. It is often conducted during or after the load tests to identify the root cause / confirm a specific performance issue. Mostly these tests are decided dynamically to debug the performance problems.