My Book on Software Performance Testing for Begineers

Here is the updated table of contents of my book.... I am happy to share that the book will be available in my blog in downdable pdf format within a week. Preview of all the chapters will be available in couple of days.

Software Performance Testing Handbook

A Comprehensive Guide for Beginners

Author : Ramya Ramalinga Moorthy

Contents

Chapter 1: Introduction.. 7
What is Performance Testing.. 7
How Performance Testing is different from Functional Testing.. 8
What is an E-Business Application.. 8
Components of a Web Site.. 9
Why Performance Testing is Crucial.. 10
Performance Testing Jargons.. 10

Chapter 2: Performance Testing – A Quick Overview.. 15
Performance Management.. 15
Performance Testing Approach.. 16
Dynamism in Performance Testing.. 18
Application Performance Management.. 19
Myths about Performance Testing.. 19
Performance Process Maturity Model.. 22

Chapter 3: Deriving Performance Test Goals.. 25
Why to set the Performance Test Goals.. 25
Know your customer.. 25
Deriving Quantitative Goals.. 26
Industrial Standards for Benchmarks.. 27
User behavior Modeling.. 27
Importance of automation in Performance Testing.. 29
Performance Test Estimation.. 29

Chapter 4: Workload Identification.. 33
What is Workload? 33
Types of Workload.. 33
Customer Behavior Modeling.. 36
Customer Behavior Model Graph (CBMG).. 36
User Community Modeling Language (UCML).. 36
Web log Analysis.. 36
Web Log Analysis Tools Overview.. 37
Web Log Analyzers Summary.. 38
Web Log Analysis Metrics.. 42
Web Site Traffic Monitoring.. 46
Overview of Statistical distributions.. 46

Chapter 5: Overview of Performance Testing Tools.. 51
Performance Test Tool Requirements.. 51
Market Tools.. 52
Open Vs Licensed Performance test tools.. 52
Criteria for choosing the Performance test tool.. 52
The most important tool.. 53
Performance Testing Tools Overview.. 53

Chapter 6: Setting up Performance Test Environment and Performance Test Scripts development best practices.. 56
Know the Test and Production environment.. 56
Test environment Isolation.. 56
Network Isolation.. 57
Load Generators.. 57
Test Data Generators.. 58
Test Execution Environment.. 58
How to choose the test scenarios.. 59
Tips for writing successful performance test scripts.. 60
Real time versus Virtual user Mapping.. 63

Chapter 7: Application Benchmarking.. 64
What is benchmarking?.. 64
Why should we benchmark the applications?.. 64
Software Benchmarking.. 64
Hardware Benchmarking.. 65
Industry Standards for Benchmarking.. 65
Transaction processing Performance Council (TPC).. 65
Standard Performance Evaluation Council.. 66
Mistakes in Benchmarking.. 67

Chapter 8: Performance Test Execution 68
Quality of Service of web application.. 68
Virtual User Simulation.. 69
Pre-requisites for Performance Tests.. 69
Types of Performance Tests.. 70
Performance Test Execution Approach.. 73
Tips for Load and Stress Test Execution.. 76
Test Execution Factors.. 77

Chapter 9: Performance Test Monitoring 81Introduction to Performance Monitoring.. 81
What Counters needs to be monitored?.. 81
Key Performance Counters.. 82
Performance Monitoring in Windows platform.. 85
Using Perfmon.. 85
Post production Monitoring.. 87
Benefits of Performance Monitoring.. 87

Chapter 10: Performance Bottleneck Analysis.. 88
Scott’s Recommendation on Scatter Charts.. 88
Scatter Chart Types.. 89
Performance Test Analysis.. 90
Best Practices on Performance Bottleneck Analysis and Isolation.. 92
Performance Testing Vs Profiling.. 94
HP Diagnostics.. 94
SQL Profiler.. 98
Know your Limits.. 99
Typical bottlenecks on DB server.. 100

Chapter 11: Performance Test Reporting.. 102
Why is it Critical.. 102
How to report the results of performance testing.. 102
Components of a good Test Report.. 103
Components of a Good Visual 104
Response Time Reporting 104
Best Practices of Test Reporting recommended by Scott Barber 105

Chapter 12: Road Ahead – Moving towards advanced Performance Engineering techniques.. 107
Moving towards Performance Engineering.. 107
What is Queuing Theory?.. 107
Types of Queuing System.. 108
What can be modeled as Queues?.. 109
Limitations of Queuing Theory.. 110
Operational Laws.. 110
a. Utilization Law.. 110
b. Little’s Law.. 111
c. Interactive Response Time Law.. 112
d. Forced Flow Law.. 112
e. Flow Balance Assumption.. 113
f. Bounding Law of Service Demand.. 113
Capacity Planning.. 114
Why Capacity Planning is required.. 114
When is it required?.. 115

Happy Learning!!!!

Performance Simulation using Ptolemy Framework made easy by PEA

The moment we start talking about Performance testing, popular market tools, type of tests, bottlenecks and the isolation techniques ,etc comes to our mind. But am sure you would be definetly surpraised to see a group who always talks about simulation , performance modeling , prediction, etc. Yes, they are none other than my favourite reference site of PEA.

Checkout this new tool available in PEA site http://www.pea-online.com/resources.htm. But be prepared for getting lost into the world of new terminolgies and jargons which might scare you little. But try understanding at a high level what is the tool all about and it be a good start to learn more about it.
Even i am very new to simulation and am a begineer to this new world of simulation. But its really interesting and and with lot of excitement am writting this note. Let me give a quick intro of all i understand.

Ptolemy is a open source project developed by group of researchers at U.C. Berkeley. It is a framework for assembling software components, a modeling and simulation tool, a block-diagram editor, a system-level rapid prototyping application, a toolkit supporting research in component-based design, a toolkit for building Java applications,etc. You can correlate it to Rational Rose (or any other modeling tool) used to create UML diagrams which consits of actors and the necessary association and aggregations. In case of Ptolemy, the models are stored in a format called MoML (Modelling Markup Language) which is created by using Vergil, Ptolemy's editor. The MoML is a modelling markup schema in XML. It is the primary mechanism for constructing models using Ptolemy framework.

In Ptolemy,it is easy to create simulation model by dragging and dropping the available actors. It provides set of actors libraries where each actor is a pre-built java class written in the ptolemy framework. You can image an actor as a web server, another actor as a DB server and interconnect them.But there will be lot of input and output parameters which needs to be defined by the user to create a successful model and it is stored as an xml file internally.

Now, it becomes too complex to drag and drop all the required components to simulate the IT system infrastrucutre. PEA has come up with composite actors which are built by using a set of available actors provided by Ptolemy. So that, we can input the required details of the server infrastructure (like, how many servers, tiers, arrival rate and service demand of various service centers,etc) through a web UI and a xml schema can be created from their tool. This will be help the user not to spend time in learning how to use Ptolemy and how to manualy feed the input to each of the actors in the model using Vergil editor. The xml file generated from the PEA tool can be fed into the Ptolemy editor Vergil to generate the text output of the simulation model. The output will be the response time predicted on the simulated server infrastructure.

In nutshell, this PEA tool takes as input deployment configuration, hardware details, workload details, etc, and generates an XML file. This XML file when run using Ptolemy (Open source, Free for use, Berkeley Lab’s Simulation Framework) can be used for simulating the performance of an IT environment. Advantage I see is the it is not required to know about simulation or its in-build actors/features to use it.

Also checkout other free tools available in PEA site. Though those analytical tools are really valuable ones, the real challenge lies in understanding how to use those tool for our system under test practically and interpret the results. Ofcourse, thats their business and formally we might need to approach them and get trained in them to use them for projects effectively.

Happy Learning!!!

Useful Info about Oracle STATSPACK Analysis

From Perfmon (Windows platform) / SAR , vmstat & other commands (UNIX platforms), you can get the insight of the server resource utilization information. But in case of the database servers, it’s very important that you need to know what is happening in the database server.

Oracle (versions later than 8i) comes with a performance diagnosis tool called STATSPACK. (In case of SQL server, try using SQL Profiler tool).

Statspack is an Oracle performance diagnosis tool (perfstat) available from Oracle 8i. It has the features of Oracle’s utlbstat.sql and utlestat.sql utilities (used in previous versions of Oracle). It has about 25 tables which help to maintain the performance information of the server.

There are good urls available in the net which details the working of Statspack. Some of them include

http://www.pafumi.net/Statspack_Analysis.html
http://www.dba-oracle.com/art_statspack.htm
http://www.akadia.com/services/ora_statspack_survival_guide.html
http://jonathanlewis.wordpress.com/2006/11/29/analysing-statspack-pt1/

There are certain things which a performance tester needs to know about the database server. I feel it’s very much required that a performance tester knows what the various terminologies (atleast) though DBA’s knowledge is not possible. From my experience, I strongly feel that Statspack analysis helps in figuring out whether something is really going wrong in the database. It would help the performance tester to answer the question - Is it worth to spend some time in tuning my database to improve system performance?

At a high-level, it works like this. Before running the test, take a snapshot (by running a simple command) of the database (snapshot collects the current value of all possible database statistics and stores it in the database table. After running the test, take another snapshot of the database and now compare these 2 snapshots (by running a simple command) and get the text report. By analysing this report, you can identify how the database was performing between these 2 snapshots (i.e... during the test run, which is what we are interested in looking at).

I am sure that if we are aware of following terms, it will be easy to figure out certain major database issues. Ofcourse, Statspack analysis performed by DBA is recommended, this is not practically achieved in many projects. In such cases, being a performance tester, you can atleast visualize bottlenecks in the database without DBA’s help.

1. Parses (hard parses vs. soft parses)
2. Transactions per second
3. User calls
4. % Library Hit & % Buffer Hit
5. Top Wait Events
6. Specific Wait events related to Indexing issues/ high table scans
7. Rollbacks per transaction
8. Top 10 SQLs sorted by Query executions, Buffer gets (CPU time) and Parse Calls.

There are lot of Statspack analysis tools available which will analyse the Statspack report and provide the analysis report. But you need to atleast know the above stated information in order to understand the report better. Try using online tools like oraperf.com or statspackanalyzer.com. If you are looking at deep insights of the database server to provide recommendations to the development team, try using tools like spAnalyzer, Wise, etc. (http://www.spviewer.com/index.html , http://www.wise-oracle.com )

Do think about Think Time

I got a chance to interact with lot of performance testers in various organizations , not only in India, but also in abroad & found that most of us are interested in spending time for setting realistic performance test goals & identifying the frequently used business scenarios (business flows) , but very few analyze the think time of various business flows.

It becomes very important that the average think time of the end users of the application needs to be analyzed rather than estimating specific time limit by counting the total fields to be filled by the end user. For the same application(lets take Yahoo new account creation), a new user might take 4-5 minutes or more to fill the required details to create a new account. But at the same time, a frequent old user, might take 1-2 minutes to create a new account. These think times (4-5 minutes for new user & 1-2 minutes for old user) might impact the server load.

Lets assume 100 new users are accessing the yahoo application for the time interval 13.00 to 14.00 hours & 100 old users are accessing the application for the time interval 14.00 to 15.00 hours continuously. Though there are 100 users (100 active sessions created) load on the server during both 13.00 to 14.00 hours and during 14.00 to 15.00 hours, the actual server load(requests) is going to be different during these 2 hours.

When you have a close look at the server load in requests/sec (Hits/sec) , you will find that the server load during 13.00 to 14.00 hours might be lesser than 14.00 to 15.00 hours, because of high think time used by the new users. If you look at the quantified numbers, you will be shocked to know that during the first hour, the average load could be apprx 12 hits/sec whereas during the second hour, the server load could be around 25 hits/sec.

Having performance test goals in mere user load alone doesnt make the test more realistic. Detailed analysis on the end user's think time makes it more realistic.

Response Time Vs Queue Length Vs Server Utilization

1. Is the system Response time directly / inversely proportional to the server utilization ?
2. Will there be queues(for CPU/Memory/Disk) if Utilization is less than 100% ?

Actually speaking, there is no correlation between system response time and the server utilization. The Response time increase might happen even when the server is less utililised. (To illustrate this, look at the ATM example available in the PEA site - http://www.pea-online.com/resources.htm).

Most of us dont understand this relationship. Increase in response time (during high load) is caused only because of queueing of the requests. The user request arrival pattern contributes to the high response time. Due to the adhoc user request arrival pattern, long queues are formed in various service centers which leads to high response time.

Response Time - What are the various components of Response time?

Waiting time (Queuing time) + Processing time = Response Time.

For a server, the processing time is always the same irrespective of load. For example, the server might take 2 seconds of processing during 1 user load and during 1000 users load for a transaction. But the waiting time would be high during 1000 users load, which drastically increases the overall response time. So, response time increases, if the waiting time increases which is caused by long queue.

Also, dont think that there will be queues only if Utilization is more than 100%. There can be queues formed even when the Utilization is less. So, everything boils down to knowing the user arrival pattern of the system. This is going to determine, when the system is going to be loaded and break.

Hence, knowing the user arrival pattern of a system is very important rather than setting mere goals to perform load/stress test. Analysing the user arrival pattern of an application would help in setting realistic goals which becomes more of organization logistics issue and most organizations bypass and pay for it later.