My Book on Software Performance Testing for Begineers

Here is the updated table of contents of my book.... I am happy to share that the book will be available in my blog in downdable pdf format within a week. Preview of all the chapters will be available in couple of days.

Software Performance Testing Handbook

A Comprehensive Guide for Beginners

Author : Ramya Ramalinga Moorthy

Contents

Chapter 1: Introduction.. 7
What is Performance Testing.. 7
How Performance Testing is different from Functional Testing.. 8
What is an E-Business Application.. 8
Components of a Web Site.. 9
Why Performance Testing is Crucial.. 10
Performance Testing Jargons.. 10

Chapter 2: Performance Testing – A Quick Overview.. 15
Performance Management.. 15
Performance Testing Approach.. 16
Dynamism in Performance Testing.. 18
Application Performance Management.. 19
Myths about Performance Testing.. 19
Performance Process Maturity Model.. 22

Chapter 3: Deriving Performance Test Goals.. 25
Why to set the Performance Test Goals.. 25
Know your customer.. 25
Deriving Quantitative Goals.. 26
Industrial Standards for Benchmarks.. 27
User behavior Modeling.. 27
Importance of automation in Performance Testing.. 29
Performance Test Estimation.. 29

Chapter 4: Workload Identification.. 33
What is Workload? 33
Types of Workload.. 33
Customer Behavior Modeling.. 36
Customer Behavior Model Graph (CBMG).. 36
User Community Modeling Language (UCML).. 36
Web log Analysis.. 36
Web Log Analysis Tools Overview.. 37
Web Log Analyzers Summary.. 38
Web Log Analysis Metrics.. 42
Web Site Traffic Monitoring.. 46
Overview of Statistical distributions.. 46

Chapter 5: Overview of Performance Testing Tools.. 51
Performance Test Tool Requirements.. 51
Market Tools.. 52
Open Vs Licensed Performance test tools.. 52
Criteria for choosing the Performance test tool.. 52
The most important tool.. 53
Performance Testing Tools Overview.. 53

Chapter 6: Setting up Performance Test Environment and Performance Test Scripts development best practices.. 56
Know the Test and Production environment.. 56
Test environment Isolation.. 56
Network Isolation.. 57
Load Generators.. 57
Test Data Generators.. 58
Test Execution Environment.. 58
How to choose the test scenarios.. 59
Tips for writing successful performance test scripts.. 60
Real time versus Virtual user Mapping.. 63

Chapter 7: Application Benchmarking.. 64
What is benchmarking?.. 64
Why should we benchmark the applications?.. 64
Software Benchmarking.. 64
Hardware Benchmarking.. 65
Industry Standards for Benchmarking.. 65
Transaction processing Performance Council (TPC).. 65
Standard Performance Evaluation Council.. 66
Mistakes in Benchmarking.. 67

Chapter 8: Performance Test Execution 68
Quality of Service of web application.. 68
Virtual User Simulation.. 69
Pre-requisites for Performance Tests.. 69
Types of Performance Tests.. 70
Performance Test Execution Approach.. 73
Tips for Load and Stress Test Execution.. 76
Test Execution Factors.. 77

Chapter 9: Performance Test Monitoring 81Introduction to Performance Monitoring.. 81
What Counters needs to be monitored?.. 81
Key Performance Counters.. 82
Performance Monitoring in Windows platform.. 85
Using Perfmon.. 85
Post production Monitoring.. 87
Benefits of Performance Monitoring.. 87

Chapter 10: Performance Bottleneck Analysis.. 88
Scott’s Recommendation on Scatter Charts.. 88
Scatter Chart Types.. 89
Performance Test Analysis.. 90
Best Practices on Performance Bottleneck Analysis and Isolation.. 92
Performance Testing Vs Profiling.. 94
HP Diagnostics.. 94
SQL Profiler.. 98
Know your Limits.. 99
Typical bottlenecks on DB server.. 100

Chapter 11: Performance Test Reporting.. 102
Why is it Critical.. 102
How to report the results of performance testing.. 102
Components of a good Test Report.. 103
Components of a Good Visual 104
Response Time Reporting 104
Best Practices of Test Reporting recommended by Scott Barber 105

Chapter 12: Road Ahead – Moving towards advanced Performance Engineering techniques.. 107
Moving towards Performance Engineering.. 107
What is Queuing Theory?.. 107
Types of Queuing System.. 108
What can be modeled as Queues?.. 109
Limitations of Queuing Theory.. 110
Operational Laws.. 110
a. Utilization Law.. 110
b. Little’s Law.. 111
c. Interactive Response Time Law.. 112
d. Forced Flow Law.. 112
e. Flow Balance Assumption.. 113
f. Bounding Law of Service Demand.. 113
Capacity Planning.. 114
Why Capacity Planning is required.. 114
When is it required?.. 115

Happy Learning!!!!

Directions for fellow Performance Testers

Before certifying the application performance,
are you satisfied with your own performance? are you sure that you are conducting effective performance test? Have you ever thought how to ensure the your performance test strategy and are you sure that your performance test report is bug free?

The following tips would help you to get more maturity and to do your job better

Yes..You need to be comfortable in the terminologies used in the performance engineering field and need to learn one or two popular and effective performance test tools and develop bottleneck analysis skills. That's the basic skills required for being a performance tester. Then you are matured enough to look back at your performance and look for how effective you are in your activities.

That's where, you need to start looking at Analytical modeling and simulation modeling approaches to move to your next level of expertise.

Foremost thing is that, understand the operational laws and start applying those laws in your day to day test activities.

Secondly start learning about Workload Modeling concepts and Statistical distribution fitting and apply those learning's to define performance testing goals/SLAs.

Thirdly start learning about Demand planning and Performance prediction techniques. Learn Mean value analysis and Erlang's method.

Fourth step is start learning about Capacity Planning.

This is the second level of maturity you need to gain. The next advanced third level is moving towards simulation modeling and advanced capacity planning.

Hope this helps to set your carrer path and plan to work towards it.

A Book on Performance Testing for beginners....

Thanks a lot viewers. Thanks for your encouragement by your kind mails.

I am planning to publish a book on Performance Testing comprising all that i learned during the past 3 years of the experience in my career.

I started my carrer as a functional tester and pushed or pulled into Performance testing fortunately thereby i came to know about this area. Mine is a hard way and i didnt had much support in terms of helping hands to get to know about the fundamendals before i started handling the projects. Slowly i learned a lot and finally ended up in unlearning a lot because of the ignorance i had initial days.

The objective of this book is to help fellow performance testers so that they can spend their quality time in achieving great things rather than reinventing the wheel :)


"Performance Testing Handbook - A Comprehensive Guide for Beginners"


The bird's eye view of topics covered in this book includes,

1. Why Performance Testing is crucial.
2. Dynamics and myths in performance testing
3. Deriving Quantitative performance test goals
4. Common mistakes in performance testing which i learned
5. Overview of performance test tools - How to choose the best tool
6. Setting up the performance test environment
7. Comprehensive support for writting test scripts
8. Application benchamarking
9. Quick look at the Industry standard benchmarks
10. A helping in setting up the performance Test Strategy
11. Performance Test Monitoring
12. Performance Test Execution - When and how to choose various tests (Load, Stress, Volume, Soak, Endurance tests).
13.Bottlenecks identification strategy
14.Reporting - How does it matters so much
15.Road Ahead - towards mathematical modeling and queuing theory basics

How to run effective performance tests

To plan for an effective performance test, follow the below four important steps

1.The first step is to understand the web site’s actual usage pattern. Mostly this information can be collected from the web server log files which are considered as the ‘gold mines’ to know every possible detail about the server load. The ‘log analyzers’, tools which helps to analyze the web log files and provide the analysis report in terms of number of user sessions, number of unique users, number of errors, peak day traffic, peak hour traffic, user request arrival pattern, frequently accessed pages, user navigation pattern, etc. From this analysis, the actual usage of the server can be identified.

2.The second step is to design the performance test approach. Based on the business forecast of the user load increase, develop service level agreement (SLAs) that has to be met by the application in order to handle the user load in the target environment. The analysis done in the first step helps to set quantitative SLA for the application. Design the test cases that needs to be tested along with the parameters like ramp up strategy, test duration, think time settings, test scenarios to be tested, different load conditions & types of tests along with the pass/fail criteria. Baseline the test approach & use this as the reference for the rest of the activities.

3.The third step is to identify the performance test tool to help us in simulating the user load on the server. Look for performance test tools available in the market (Licensed or freeware tools) based on the budget constraints & technologies supported by the test tool. Using the test tool of choice, develop the test script which simulates the actual user access pattern. Have enough test data available in the test script in order to simulate different users using different data available in the script.

4.The fourth step is to look for the readiness of the test environment. The test environment includes the application’s server infrastructure & also the load test environment. Enough care should be taken to ensure that no other users access the application during the test run & the servers are isolated from any other usage. In the case of load test environment, ensure that load generators are in place to generate the required load, the availability of test tool license to support the load to be simulated, test scripts are developed and ready with the required test data & think time, server monitors are configured with the required performance counters & the server admins are informed for their assistance in case of server failures.

Choose right set of test scenarios to schedule the performance test

Like functional testing, performance testing cannot be done for all scenarios. Identify the frequently used scenarios based on discussions with the business analyst. Sometimes, there will be couple of scenarios which have some suspections about its performance. We have to include those kind of scenarios too. It boils down to Pareto analysis – the system performance will be improved if 20% of the scenarios which are most frequently accessed are tuned. Tuning the remaining 80% of scenarios will not bring considerable performance improvement. Use Scott barber’s UCML or CBMG modeling to identify the critical scenarios.

During the performance test, don’t forget that you need to create the actual real time system usage model. (i.e.) The test should be very realistic in such a way that it models the real time traffic & more important is it should model the transaction mix of the real time workload. If you are scheduling a test for 100 users, then make sure that the 100 users are distributed across various functional flows of the application (Say in a banking application during the peak hours the users are distributed across various functionalities like, 60% of the users depositing the amount, 20% of the users transferring the amount, 10% of the users checking for the account balance & 10% of the users accessing the mini-statement). Unless this kind of mixed mode scenario is adopted for the testing, we might not end up simulating the real time user load.

There are certain situations wherein we might need to schedule the performance test with a single functional flow. One situation is when the performance testing is scheduled for each of the functionally stable build. Other situation would be when we identify some performance issues during the mixed mode scenario, then in order to isolate whether the performance issues are from a specific functional flow, we might need to run single functional flow test.

Observe the differences quoted above. As long as your objective is to test the system performance for a specific load, make sure the load is very realistic to the real time user access.

Metrics to be collected during any Performance Tests

Irrespective of the kind of performance test (load test/stress test/volume test/spike test etc), there are certain metrics which needs to be collected from the performance test tool to analyze the system performance.

End of the day , we want to know whether the system is able to handle the load meeting the response time level SLA. To achieve that we should identify the following 2 things.

1. Whether my software pipe is able to effectively utilize the available hardware pipe.

2. Whether my hardware pipe is big enough to be used by the software pipe.

If the software pipe is not able to utilize the hardware pipe effectively, then no wonder that even if you upgrade the server hardware, there will no improvement in the system performance.

Metrics to be collected

1.As a first step always look for the average & 90th percentile response time of all the transactions. Just neglect if there is other transactions which are inserted by the tool itself. (In case of Load Runner, the tool provides the transaction response time for vuser_init, vuser_end & action transactions depending upon the run time settings).
2.Next look for the load generated from the test tool. Hits/sec is the metric which provides the information on how many requests are fired to the server per second. Look for the average hits/sec & if the test is run using a specific ramp up rate, then measure the hits/sec value during the stress period of the test(when all users where running).
3.Next look for the Throughput, the metric which portrays the load handled by the server. The total transactions passed per second metric helps to identify the how capable is the server to handle the transactions. Also look for throughput in terms of bytes/sec metric which again portrays the server capability to respond to the client requests.
4.Next look for the failures occurred during the test. Observe when the failures have occurred, what is the type of failures (network failure, HTTP 500 errors, HTTP 404 errors, proxy server failures, etc) & is the failures specific for a transaction.
5.Next, look for the basic performance counters of web/application/DB server resources like %CPU, processor queue length, Memory pages/sec, Page faults/sec, % Disk utilization.

These five are the basic performance metrics which are mandatory for any performance tester to analyze the performance of the system. Going forward, in order to isolate the performance bottlenecks, we might end up looking at various performance metrics depending upon type of the performance issue.

Know the importance of Think time

In most of the performance testing approach, the ‘Think Time’ metric is taken for granted. We should understand the importance of think time before deciding the test strategy.

What is a Think Time? Think time is the time taken by the users to think or to navigate
to different pages in the application.

Depending upon the application context, the think time would vary. Its not advisable to have the default think time for all applications under testing. For the user registration page containing 5 mandatory fields, the use might take 15-20 seconds on an average. Notice the words “15-20 seconds on an average”. You can’t say that all the users would take exactly a specific think time. It would definitely vary based on each of the users. So, always make a point that whenever you run the load test, apply a randomization factor on the configured think time. Say if the think time in the script is 10 seconds between each of the transactions, configure the load test tool to use a randomized think time of 80% to 120% of the value available in the script.

As we are interested to measure the response time of the transaction (user defined actions), care should be taken to place the think time outside the transaction points. (i.e.) the think time should not be placed between the start & end of the transaction.

Most of the testers doing the performance testing don’t understand that by changing the think time, we can simulate the load of large number of users though we run the test for less number of users. To illustrate, consider the following example. Five tests were conducted with 5 virtual users simultaneously accessing the application. In each of the test, the think time is the only metric which was changed. Observer, the Hits/sec generated from the client side & Throughput (the number of transactions passed /sec).

Though the number of users hitting the server is same, the think time creates a huge impact on the server load. We need to understand the fact that decreasing the think time will increase the server load & vice versa. Always try to model the real time scenario & identify the performance figures. Never forget that we can play around with the server load just by changing the think time.

Also, dont forget to use Little's law, N = X . (R+Z) to check whats the actual load on the system for the test which you are running.