Learn To Win

Performance Testing ebook details

2010-12-09T11:28:00.006+05:30

Dear Viewers,

Would like to send out my thanks message for all of you for all your appreciations for my first book (Version 1) : Software Performance Testing Handbook - A Comprehensive guide for beginners. Am happy to share with you that the book has crossed 2000 downloads till now. Going forward the ebook will not be available for free download. You can purchase this book online in the below link : http://i-proclaimbookstore.com/sopetehacogu.html.

Your appreciations has motivated me to work on my second ebook (Version 2) : System Performance Testing - Performed Rightly. This ebook is half way through and i am planning to publish it by April 2011.

Info about New Ebook on Performance Testing

2010-11-17T10:18:00.005+05:30

Thank you very much for your overwhelming response and mails for my ebook - Software Performance Testing Handbook - A Comprehensive Guide for Beginners.

Your overwhelming response is the sole inspiration for me to write my Version 2 of my book - System Performance Testing - Performed Rightly.

Looking forward for your continuous support ad interest!!

Happy Learning!!

Note :
My first version of the ebook can be purchased here :
http://i-proclaimbookstore.com/sopetehacogu.html

WCF Services Performance Testing

2009-04-20T22:06:00.002+05:30

Most of the market available performance testing tools available for web services performance testing does not provide support to WCF (Windows Communication Foundation) web services performance testing.

The following are the tools which supports WCF web servcies performance testing.

> WCFStorm
> WCF Load Test
> SOAtest

SOAPUI doesnt support WCF web services performance testing. Out of above 3 tools, we have finlized on the SOAtest 5.5 tool to carryout WCF web services performance testing.

Web Services Performance Testing

2009-04-20T21:29:00.003+05:30

Web Services Performance Testing using SOAPUI

Recently i had a chance to recommend a performance testing tool to carryout performance testing for web services system. I evaluated a series of open source and licensed tools and finally concluded SOAPUI would be meet the requirement for our system.

soapUI provides extensive load testing functionality allowing you to do the following:

Functional LoadTesting : validate functionality under load using standard TestCase methods.

Behavioral LoadTesting : analyze performance behaviour under varying load with different load strategies.

Performance LoadTesting : find maximal performance available using thread strategies and Command Line LoadTest execution.

Requirements Driven LoadTesting : define performance requirements and continuously validate using Load Test assertions.

SOAPUI helps in the following.
> Validate a Web Services performance under different Load scenarios.
> Maintain Functional validations to see that they don't "break" under load.
> Run several load tests simultaneously to see how they affect each other.

Its a freeware tool which best suits web services load testing. Its commercial version is called SOAPUI Pro includes productivity enahancements.

In Overall SOAPUI meets most the requirements for a Web services performance testing and it is a apt tool to meet most of the web services.

Software Performance Testing Handbook - A Comprehensive Guide for Begineers

2009-03-30T21:10:00.012+05:30

Dear viewers,

Hereby am publishing my first version of ebook on performance testing which was pending for a long time :). I have tried my best to compile my learnings and experience to help you out. I guess you will get benefitted out of it.

Happy Learning!!!

Chapter 12: Road Ahead – Moving towards advanced Performance Engineering techniques

2008-12-02T19:15:00.006+05:30

Software Performance Testing Handbook

A Comprehensive Guide for Beginners

Moving towards Performance Engineering

As discussed in earlier chapters, mere plans for Performance testing during the end of the SDLC cycle is considered as a reactive approach wherein Performance Engineering approach aims at integrating the performance engineering activities from starting of the SDLC to build a system with high performance. Thus, Performance testing forms a subset of the Performance Engineering activity. Accordingly, the competencies required for a performance engineer is little different from the performance tester.

Software Performance Testing is the act of evaluating the software system for its performance and finding the bottlenecks in the system. Software Performance Engineering is the systematic approach of constructing the software that meets the performance objectives.

Nowadays, IT companies have slowly started adopting the performance engineering activities for critical applications though 70-80% of them have adopted mere performance testing activities.

What is Queuing Theory?

Queuing Theory is the mathematical study of waiting lines (or queues). It is interchangeably spelled as Queueing Theory or Queuing Theory. It is considered as a branch of Probability Theory and Operational Research as it is used in making business decisions. It is used in various fields like telecommunications, networks, traffic management, etc. The notation for describing the queuing model was first introduced by David .G. Kendal in 1953.

The figure represents a model of a single queue. Jobs arrive at the queue from the input source, each job bringing a demand for service from the queue. If the queue becomes full, the arriving job is not allowed to join the queue. If the job enters the queue, it may compete with other jobs for service from the queue. When a job completes its service, it departs the queue. We identify models in this class by describing the input source (how jobs arrive), the demand jobs that it brings with them, the rules by which they compete for and receive service, and the capacity of the queue. Thus the elements of the queue includes the input process, service mechanism, queue discipline and output process.Queuing theory provides answers like mean waiting time in queue, mean response time, service time (service demand), user request arrival pattern and the distribution of users in the system, etc.

Types of Queuing System

The Queuing system can be classified into 2 broad categories as Open or Closed System.

A system can be classified as Open or Closed based on certain characteristics it possesses.

Open System

These systems are usually internet facing where the request jobs arrives the system from outside externally and the arrival pattern of the requests are independent of the way the system processes the incoming requests. Thus the load on the system at any point of time is the external decision. The server workload for such systems can be represented as arrivals per unit time.

Closed System

These systems are usually client-server systems where the behavior of incoming request jobs are influenced by the server processing mechanism. The requests circulate continually forming a closed loop. Normally users fire a request, get a response from the server and think for certain amount of time (think time can be zero or non-zero value) and then again fire the request. Thus the load on the system at any point of time is decided by the system. The server workload for such systems can be represented as the number of users and think time.

Operational Laws

The operational analysis is a set of basic quantitative relationships between performance quantities. The operational analysis is used to establish relationships among quantities based on measured or known data about computer systems.

Capacity Planning

Capacity planning is the process of determining what hardware and software configurations are required to adequately meet the application needs. It helps to define the hardware and network infrastructure needed to handle the expected and forecasted server load. Thus the goal of capacity planning is to provide satisfactory service levels to users in a cost-effective manner.

Capacity planning is carried out by measuring the number of requests the server currently processes and how much demand each request places on the server resources. Using this data, the resources (CPU, RAM, disk space, and network bandwidth) necessary to support current and future usage levels can be calculated.

Why Capacity Planning is required

The first and foremost reason is the user experience factor. Consider a system that can support only a given number of concurrent users while guaranteeing a reasonable response time. As many of us have experienced, when traffic increases on a major Web site that isn't adequately equipped to handle the surge, the response time deteriorates significantly. Am sure that many of us have experienced situations where sites, say payroll, when traffic increases (during last day of the month) on the site, the site isn’t adequately equipped to handle the sudden spike and hence resulting in response time deterioration. Studies have shown that if a site's response time is more than 10 seconds, then end users tend to leave. This is generally a bad thing and should be avoided, as there is no secret that a web site's downtime can result in a significant amount of loss to the business.

The second reason is that capacity planning helps you to decide how to allocate resources for a system in terms of the CPUs, RAM, Internet connection bandwidth, and LAN infrastructure needed to support the required performance levels and plan for future growth. Once we understand the limitations of the existing hardware configuration, we can estimate the amount of additional hardware needed to support any increased demands in performance.

Finally, capacity planning is important because it helps to answer the question of what hardware and software infrastructure is required to enable the current system to achieve expected performance objectives.

Chapter 11: Performance Test Reporting

2008-12-02T19:12:00.001+05:30

Software Performance Testing Handbook
A Comprehensive Guide for Beginners

The Performance Test Reporting approach is totally different from the functional test reporting way. In the Performance Testing world, it is not easy to provide the pass/fail criteria as in the case of functional test report. The performance test report development is not as easy as everyone thinks because the test report should contain the symptoms and observations of the performance bottlenecks along with the recommendations. More than that, the information should be abstracted based on the audience of the test report.

Why is it Critical

The Performance Test Report is the document or presentation which contains the details of the performance test outcome. In case of business critical systems, the Performance Test Report would have a direct impact on the business decisions regarding the production release of the application. In some cases, the decisions related to infrastructure upgrades depend upon the test recommendations provided in the Performance Test Report. In case of products, few sections of the Test Report can also shared with all the clients as the performance benchmark. Though we should accept the fact the test report contents purely depends on the test objective, there are certain basic fundamental things which needs to be followed to develop a good test report.

How to report the results of performance testing

The foremost information to be provided in the Performance Test Report should be the overall objective of the performance testing. It should also provide the overview of the application and the business scenarios identified for testing and the user distribution model used for the testing. Though these details are provided in the test plan document, it is a good practice to include these information in the test report.

The main objective of the Performance Testing is to identify the server capacity (Throughput) to handle user load. The business might also be interested in the response time identification for the critical transactions. The Performance Testing activity provides the insight of SPEED, SCALABILITY and STABILITY of the application. The Performance Test Engineer should have this in mind while writing the test report.

The important information that should be available on the Performance Test Report includes the response time details in seconds, server throughput details in terms of transactions per second or Bytes per second , application errors or other errors encountered during the test and an indication about the hardware resource utilization levels.

Mere copying of few graphs from the performance testing tool doesn’t meet the objective of test report creation. The Performance Test Engineer should explain the test results through plain sentences in business English about what is the inference made from each of the graphs provided in the test report. In case of specific bottlenecks, symptoms needs to be explained as all the readers of the test report may not understand from looking at the graph. Each of the graphs should have its purpose detailed along with the clear details of x-axis and y-axis.

The Performance Test Engineer should ensure that right statistics are used and the report is customized to the intended audience. Also the performance test report is expected to have more visuals rather than providing the test results in text format.

Components of a good Test Report

A Performance test report will have different kind of readers like business people (management level people), technical members (architects, designers) and infra members (server admin and DBA). A good test report should meet the expectations of different kinds of readers and should make them understand the test results by adopting right level of abstraction.

It is always a best practice to have a separate section in the test report for providing the test results summary with a high level abstraction of information where technical details needs to be presented at high level but enough information on the test conclusions should be available as major business decisions are made based on this summary.

At a high-level, a performance test report should have the following information readily available.

• Performance Test objectives / goals (Service Level Agreements)
• Test environment and the server configurations
• Tests conducted (along with the deviations from the plan)
• Performance Test Result summary
• System Performance details with enough visuals
• System Performance Bottlenecks details with enough visuals
• Performance test conclusion and recommendations

Chapter 10: Performance Bottleneck Analysis

2008-12-02T18:59:00.007+05:30

Software Performance Testing Handbook

A Comprehensive Guide for Beginners

Often Performance Test Engineers feel performance bottleneck analysis is a complex task. Unless you experience bottlenecks and experiment few things, one cannot become an expert in bottleneck analysis by reading articles or books. I agree that it is more of an art than science. But bottleneck analysis and isolation can be made easy when systematically approached and fundamental concepts in Queueing Theory are understood. This chapter explains few things which every Performance Test Engineer needs to have in mind during performance bottleneck analysis.

Scott’s Recommendation on Scatter Charts

Scatter Charts are one of the powerful tools for bottleneck analysis. It provides a quick view of the bottleneck and it is easy for a Performance Test Engineer to explain it to non-technical stakeholders.

As part of the bottleneck analysis, the first chart which every Performance Test Engineer needs to be look for is the response time chart. Look for the trend in the server processing time for each of the transaction or timer provided in the script during the span of the test. Always ignore the metrics collected during ramp up and ramp down as any conclusion should not be made while the server load is changing. Any metric that is collected during the stable constant load period can only be considered for the analysis.

A Scatter Chart can be created by having test time plotted in seconds on X-axis and the response time measure plotted in seconds on Y-axis. Each measurement represents the server response time for a specific transaction or timer provided in the script during the span of the test. A simple scatter chart is provided below. The blue dots represent the response time for loading the system home page and the red dots represent the response time for login transaction.

Performance Test Analysis

Though the performance testing activity is to simulate high loads on the system and to identify the system break points, practically not all the projects will have the objective to find bottlenecks on the system due to time or cost constraints. Many a time it so happens that project stakeholders expect the Performance Engineers to do just an application benchmarking (run the planned tests and report the system behavior), though there is a planned production move after the performance test. In my experience, many a time it happened that I end up convincing the project stakeholders about the business risk in doing application benchmarking without planning for bottleneck analysis and isolation. I have to accept that sometimes I have failed in my attempt and ended up doing benchmarking in spite of project being planned for production move.

It is very essential that every Performance Test Engineer should understand what is required for the application and suggest it to the stakeholders irrespective of limiting to time or cost constraints. However, acceptance is under stakeholder’s discretion.

Whenever any performance tests are run (benchmark tests, load tests, stress tests, etc), the first metric which needs to be looked at is the response time metric. This is the basic metric which needs to be checked to know the server’s processing time, time taken by the server to respond to user requests. Always adopt the practice of merging response time and running users graphs both for analysis and reporting purpose. Then look for the scatter chart pattern and start looking at the suspects.

The second important metric to look at is the server throughput, server’s processing capability in terms of requests handled per unit time or transaction handled per unit time or amount of bytes returned to the users per unit time. For this metric, there are several graphs available in the test tool like Hits per second, Transactions per second, Throughput (in Bytes/second).

I am sure you will not accept why I am talking about hits per second metric as this metric refers to the input load provided to the system and not the server’s output. To understand it better, consider this example. There is a small tank which has an inlet and outlet pipe of same capacity at the top and bottom of the tank respectively. Assume the tank doesn’t have any other leakage; Fill half of the tank by water having outlet pipe closed and then open the outlet pipe and observe the water inflow and outflow. We can observe that the rate of water flowing into the inlet pipe will be the same as the rate of water flowing out of the outlet pipe. This is true as long as the tank is in good stable condition. The same applies to the server during stable state. The incoming flow rate, A (arrivals per unit time) and out coming flow rate, C (completions per unit time) will be equal during the stable state (as long as there is no saturation in the server).

A = C where A is the arrival rate & C is the completion rate

The Hits per second graph provides the details on the requests handled by the server during unit time. For increasing loads, the hits per second should linearly increase and during the constant load period, the hits on the server should be constant. If there is a drop in the hits per second value, then it represents an issue on the server. This represents that the server is not able to handle the incoming requests which represents instability in the server. This issue needs to be cross verified with the server monitoring graphs to look for saturation issues of various service centers.

The errors observed during the tests needs to be analyzed and test data related issues needs to be isolated from the application errors. If test data related issues exist, it needs to be fixed and the test needs to be rerun to check for consistency in the issues.

Performance Testing Vs Profiling

The performance test tools have the capability to pinpoint the user actions which have high response time. By configuring right set of monitoring on the server infrastructure, hardware related bottlenecks could be effectively identified during performance tests. In the case of software bottlenecks, the performance test tools in general does not provide the details like which tier or software component consumes more processing time. Also, the performance test tools do not have the feature to drill down to the method call or component which contributes to high response time. Any issues related to the application code cannot be identified from the performance test tool.

Profiling is a dynamic program analysis which helps in understanding the program behavior by inserting specific code into the program code (instrumentation) to capture the call times, call stack, frequency of method calls, call performance, thread concurrency issues, memory usage limits, garbage collection details and heap behavior during the program execution. The profiling tools hooks into the JVM of the application (Java) and instruments the application classes to monitor its performance hence provide capabilities to show the call chains and helps in identifying the line of code which leads to high processing time for 1 user load. Profilers are considered as unit test tools as it helps in identifying code performance and it would be a best practice to do profiling and address the software issues before subjecting the system to performance tests.

HP Diagnostics

HP Diagnostics tool facilitates drill down capabilities to identify the root cause for high response time transactions of the system. It helps to identify slow performing components, memory leaks, thread contentions, slow SQL query, slow layers and concurrency issues during the time of the load tests. The ability to provide the server performance behavior during the loaded condition makes HP Diagnostics tool special and different from other profiler tools like Dev Partner, Glow code, etc. The Diagnostics tool can be integrated with the HP performance test tools (Load Runner, Performance Center), the same performance test scripts used for performance testing could be used to create the required load on the server and the server performance characteristics could be monitored and diagnosed.

Chapter 9: Performance Test Monitoring

2008-12-02T18:55:00.003+05:30

Software Performance Testing Handbook

A Comprehensive Guide for Beginners

Introduction to Performance Monitoring

Performance monitoring is the process of collecting and analyzing the server data to compare the server statistics against the expected values. It helps the performance tester to have the health check of the server during the test. By monitoring the servers during the test, one can identify the server behavior for load condition and take steps to change the server behavior by adopting software or hardware performance tuning activities.

Each performance counter helps in identifying a specific value about the server performance. For example, % CPU Utilization is a performance counter which helps in identifying the utilization level of the CPU.

In a nutshell, server monitoring should provide information on four parameters of any system: Latency, Throughput, Utilization and Efficiency, which helps in answering the following questions.

Is your server available?
How busy is your CPU?
Is there enough Primary Memory (RAM)?
Is the disk fast enough?
Is there any other hardware issues?
Is the hardware issue result of software malfunctioning?

Key Performance Counters

Always start with few counters and once you notice a specific problem, start adding few more counters related to the symptom. Start monitoring the performance of the major resources like CPU, Memory, Disk or Network. This section provides you the details about key counters from each of above mentioned 4 areas which are very important for a Performance Tester to know.

Processor Bottlenecks

The bottlenecks related to processor (CPU) are comparatively easy to identify. The important performance counters that helps in identifying the processor bottleneck includes

% Processor Utilization (Processor_Total: % Processor Time) – This counter helps in knowing how busy the system is. It indicates the processor activity. It is the average percentage of elapsed time that the processor spends to execute a productive (non-idle) thread. A consistent level of more than 80% utilization (in case of single CPU machines) indicates that there is not enough CPU capacity. It is worth further investigation using other processor counters.

% User time (Processor_Total: % User Time) – This refers to the processor’s time spent in handling the application related processes. A high percentage indicates that the application is consuming high CPU. The process level counters needs to be monitored to understand which user process consumes more CPU.

% Privileged time (Processor_Total: % Privilege Time) – This refers to the processor’s time spent in handling the kernel mode processes. A high value indicates that the processor is too busy in handling other operating system related activities. It needs immediate attention from the system administrator to check the system configuration or service.

% I/O Wait (%wio - in case of UNIX platforms) – This refers to the percentage wait for completion of I/O activity. It is a good indication to confirm whether the threads are waiting for the I/O completion.

Processor Queue Length (System: Processor Queue Length) – This counter helps in identifying how many threads are waiting in queue for execution. A consistent queue length of more than 2 indicates bottleneck and it is worth investigation. Generally if the queue length is more than the number of CPUs available in the system, then it might reduce the system performance. A high value of % usr time coupled with high processor queue length indicates the processor bottleneck.

Other counters of interest:

Other counters like Processor: Interrupts per second, System: Context Switches per second can be used in case of any specific issues. Interrupts per second refers to the number of interrupts that the hardware devices sends to the processor. A consistent value of above 1000 in Interrupts per second indicates hardware failure or driver configuration issues. Context Switches refers to the switching of the processor from a lower priority thread to a high priority thread. A consistent value of above 15000 per second per processor indicates the presence of too many threads of same priority and possibility of having blocked threads.

Performance Monitoring in Windows platform

Windows operating system comes with a performance monitoring tool called Perfmon. This monitoring tool can be used to collect the server statistics during the performance test run. Normally, most of the performance testing tools have its own monitors to monitor the system resources of the system under test. In this case, it becomes easy to compare the system load related metrics with the system resource utilization metrics to arrive at a conclusion. Infact, any performance testing tool that monitors windows machine internally talks to perfmon to collect the system resource utilization details.

Post production Monitoring

There are lots of licensed tools available in the market which is used for post production monitoring. These tools are used to capture the web server traffic details and provide online traffic details. A very popular tool of this category is WebTrends. Many organizations uses this tool to get to the traffic trends of an application running in production environment. There are other tools like HP OpenView tools which run in production servers and monitor the server resource utilization levels. It provides alarming mechanism to indicate the heavy usage and provides easy bottleneck isolation capabilities. But due to the cost involved with these kinds of tools, most of small organizations don’t opt for them. But post production monitoring data would be of great use for designing realistic performance tests.

Benefits of Performance Monitoring

· Allows you to analyze and isolate the performance problem.

· Understand the resource utilization of the server and make best use of them.

· Plan for Capacity Planning activity based on the resource utilization level.

· Provides the server performance details offline (by creating a alert of sending a mail/message when resource utilization reaches the threshold value).

Chapter 8: Performance Test Execution

2008-12-02T18:49:00.003+05:30

Software Performance Testing Handbook
A Comprehensive Guide for Beginners

In the Internet, the competitors are only a mouse click away. Hence the web sites should provide high performing, time effective application thereby increasing the satisfaction level of the end users. Dissatisfied users would abandon the site and never come back. Bad mouthing would spoil other user’s interest towards the web site. Hence there is a high necessity to design applications with high performance to meet the expectations of the end user.

There are certain activities that need to be performed before we plan for the performance tests for a web site. By ensuring these activities, we could improve the accuracy of the performance tests and would run tests simulating the real time usage.

Quality of Service of web application

The QOS (Quality Of Service) of a web application is normally measured using the system response time and system throughput. It is about ‘How fast is the system able to respond to the customer requests?’ and ‘How much load the server can withstand by providing quick response to the customers?’ There is a high chance for losing the business when the customers are not satisfied with the QOS of the web site.

Providing QOS to the end users of the system is a big challenge faced by the application designers. The objective of the system testing is to identify the functional defects before the end user finds it. The system needs to be assured not only for its functional stability but also for its availability and responsiveness at all times of the user load. The end users will not be happy just by having a functionally stable system; the system should be very responsive and available enough at all times of user accesses.

It is very important for a web application to respond quickly with minimal processing time irrespective of number of users accessing the system. Though the application might be functionally stable for 1 user, but there could be inherent design issues or code level issues which might create show stopper issues during multi user load. Hence it is highly required to test the system performance before releasing the system to the open community.

Types of Performance Tests

There are various types of tests that could be performed on an application during the performance certification activity. The objective and importance of various types of tests are discussed below.
1. Baseline test
2. Benchmark test
3. Load test
4. Stress test
5. Soak test
6. Endurance or Longevity test

But it is not required that all application needs to undergo all of these tests during performance testing. It purely depends on the application context and its usage pattern. It is always recommended to carry out any special tests which are different from those mentioned above, if the application usage pattern demands one.
Performance Test Execution Approach

Let us recollect the basic objective of running a performance test. The Performance test is conducted to identify and certify how fast the application responds and at what point of load the application performance starts degrading. The very important expectation on the performance test activity is the ability to run more realistic tests. By injecting the required number of users load during the test, realistic end user usage pattern cannot be achieved. The server workload needs to be analyzed which is purely based on the expected accuracy level in the test results. For a performance critical application, often high accuracy is expected combined with high level of confidence about the system performance.

Normally, for performance testing, paretto’s 80-20 rule should be followed. After identifying the most frequently used business flows, user distribution among the identified business flows needs to be done carefully. Though 100% real time simulation is impossible, lot of analysis should go into this analysis to come up with more accurate realistic numbers. Usually for this analysis, web log analysis provides lot of inputs in case of applications where usage history exists. A high accuracy level can be expected in case of the availability of real time usage data. For applications, going to production for the first time, expected accuracy level should be less due to the non-availability of real time data.
It is always better to neglect those business flows which are seen or expected to be occasionally used during peak business hours. Rather during performance tests, subject the server to a constant load for 15-30 minutes approximately (also depends on application usage pattern) to get more samples of server behavior (with respect to response time and resource utilization levels) during loaded condition and then certify the application for the subjected load. A test could be called realistic only when the user load and the transaction mix are appropriate along with the associated confidence about the server performance behavior.

It is required to verify the system performance for more number of samples and then conclude the performance metrics for response time, server utilization levels, etc. The test should always have sufficient time for user ramp up, before subjecting the system to the expected load period. All the measurements about the server performance need to be taken only during the stable load period. The test should be configured in such a way that all users are actively accessing the system during the stable load period. If the peak expected server load is for example 200 requests per second, then the test should be scheduled in such a way that that running users should create a constant load of 200 requests per second atleast for 10 to 15 minutes duration. Though the peak expected user load (200 requests per second) occurs for less number of times in the real time with sufficient time interval between the spikes, the performance test should subject the system for a constant load to get more samples on the server performance. This helps in assuring the server performance behavior for a peak traffic situations. Hence the Performance Test Engineer should come up with a proper test strategy in order to test the system for realistic load and also at the same time, the test should be able to guarantee the system performance behavior for the peak load situations.

Let us discuss an example to validate the load test scenario created for the business requirements. The application needs to be validated for the 100 user load which consists of following 3 business scenarios.

Business Scenario A -60% of target load
Business Scenario B -30% of target load
Business Scenario C -10% of target load

Based on the inputs from business analysis, it was found that scenario A will be executed 6 times in an hour, scenario B will be executed 2 times in an hour and scenario C will be executed 1 time in an hour. Also, it was known that the peak server load was 200 requests per second during the peak hour.

The following load test is scheduled in order to meet the above test requirements.

This load test strategy meets the user load targets (100 users accessing the system for 40 minutes of stable load period) and the number of executions of each business scenarios (6 iterations of A scenario, 2 iterations of B scenario and 1 iteration of C scenario). During the stable load period, when all three business scenarios were running (40:00 to 50:00 mm:ss), the server load created on the server was about 200 requests per second. Hence the test seems to be realistic to the production load.

The above test scenario meets the performance test requirements of the applications and it provides the system performance figures for the realistic system usage. But what is missing in the above load test scenario is the confidence level of server being able to handle the 200 requests per second, as there was only 1 sample collected during the interval (40:00 to 50:00 mm:ss) when all three business scenarios were running. A load test should aim for assessing the server performance for realistic usage pattern and also create a high confidence by analyzing the system performance for atleast 3 to 4 samples. The confidence level increases only when more number of samples is made available and analyzed for server behavior. Also, the server resource utilization levels needs to be assured only by subjecting the system to a stable constant load. Hence it is recommended to run all the three business scenarios together for atleast 20 to 30 minutes to collect enough samples on the response time metrics and server resource utilization metrics by maintaining the load at 200 requests per second. This can be combined in the same load test or can be run as a separate test depending the application performance requirements and performance behavior of the system.

The moment we talk about realistic tests, think time should be the high priority one worth spending time for detailed analysis. In spite of having correct business flows and user distributions between the business flows, the moment think time is not realistic, the entire test goes for a toss leading to unrealistic tests. The think time decides the user request arrival pattern of an application which decides the load on the server at any point of time. Usually it is always recommended to use realistic think time between transactions and randomize it while running the test (say 80% to 120% of configured think time). This randomization helps is simulating the think time of inexperienced user of the web site and the frequent user of the web site.

Chapter 7: Application Benchmarking

2008-12-02T18:46:00.001+05:30

Software Performance Testing Handbook

A Comprehensive Guide for Beginners

What is benchmarking?

In general, Benchmarking is the process of determining who is very best, who sets the standard, and what that standard is. When it comes to the Performance Testing context, most of the time we often use the word benchmark. Benchmarking is the process of the determining the relative performance of the software program by running the standard set of tests. We can benchmark the hardware or software performance by comparing its relative performance figures. For example, we can benchmark the software application across various application servers by deploying it in Websphere server, Weblogic server and JBoss server to compare the performance of the application in each server. We can benchmark the hardware requirement for the software by running the tests on Dell server, HP Proliant server and IBM servers to compare the CPU, Disk and Memory utilization details.

Why should we benchmark the applications?

Unless we compare ourselves with others against a common measure (for example say height) which is a measure applicable for both entities, we cannot say who is best. It becomes very tough if we don’t know what is the standard measure that can be used to compare oneself against the other. The same is applicable for the software systems. One always needs to know where their competitors stand. In order to compare different software or hardware performance, we need to have the common standard measurements.

Industry Standards for Benchmarking

It becomes impossible for the application owners to test the application performance against variety of the server machines available in the market and to choose among them due to cost factor. There are organizations available in the market which does this benchmarking. The application owners could refer to these industrial benchmarks to decide on their infrastructure requirements. Transactions Processing Performance Council (TPC), Standard Performance Evaluation Council (SPEC), Synchromesh benchmarks, etc are the industry standard benchmarks available. There are other open source and vendor specific benchmarks available in the market. These organizations perform the testing on different servers with varied hardware configurations and provide the performance figures. As we all know we need to have some common measure to compare with the competitors, these industry standards provide the list of measures which is common across server platforms.

Transaction processing Performance Council (TPC)

The TPC is a non-profit corporation founded in 1980’s to define transaction processing and database benchmarks and to disseminate objective, verifiable TPC performance data to the industry. The TPC benchmarks are widely used today in evaluating the performance of computer systems.

The TPC benchmarks involve the measurement and evaluation of computer functions and operations through transaction as it is commonly understood in the business world. A typical transaction, as defined by the TPC, would include updating to a database system, a set of operations including disk read/writes, operating system calls, or some form of data transfer from one subsystem to another. There are different types of benchmarks available in TPC. It includes TPC-App, TPC-C, TPC-E and TPC-H. The following information is taken from the TPC site - http://www.tpc.org/.

Standard Performance Evaluation Council

The SPEC is a non-profit organization that aims to produce fair, impartial and meaningful benchmarks for the computers. SPEC was founded in 1988 and the goal is to ensure that the marketplace has a fair and useful set of metrics to differentiate candidate systems. Its member organizations include leading computer and software manufacturers. SPEC benchmarks are widely used today in evaluating the performance of computer systems; the results are published on the SPEC web site - http://www.spec.org/ .

Mistakes in Benchmarking

One needs to understand the fact that benchmarks provided by organizations like TPC, SPEC are based on the workload created by a typical application used for the performance testing. If the application owners develop applications with a specific workload which is totally different from the typical application workload used for benchmarking, then it is not an apple to apple comparison. But we don’t have any choice as it is impossible to test our product across different configurations to choose the best one. Hence we can refer to the benchmarks and add appropriate risk buffer if the workload of the application under test is completely different from the one used for benchmarking.

Chapter 6: Setting up Performance Test Environment and Performance Test Scripts development best practices

2008-12-02T18:39:00.003+05:30

Software Performance Testing Handbook

A Comprehensive Guide for Beginners

Know the Test and Production environment

The actual target production environment of the system under test needs to be studied by the Performance Test Engineer. The performance test strategy should contain the details about the target system environment on production. The Performance Test Engineer should know the deployment architecture of the application in the target production environment and must educate the development team to setup a similar test environment to run the performance tests. There could be a huge difference in the system performance on test and production environment as both environment uses different hardware platform to deploy the application. For example, if a three tier application deployed in a test environment gets deployed using two tier architecture in the production environment, then the system performance would have a major difference in both the environments.

Always identify the system configuration details of the server machines in the target environment like the number of CPUs, CPU capacity (clock speed), RAM capacity, disk capacity, free space available in the disk, NIC card capacity and network bandwidth. These details needs to be identified before scheduling the performance test and should be documented in the test plan document for future reference.

Test environment Isolation

If the performance test bed is setup locally, then the test bed should be isolated from the other test beds used for functional testing, UAT (user acceptance testing), etc. The Performance test should be conducted in an isolated environment, so that it is easy to isolate the bottleneck and avoid unnecessary extra load during the performance test. Also there is a high probability that servers could create major failures during load conditions which might affect other activities if planned on the same environment.

Network Isolation

Isolating the network used for the performance test is very important if the objective of the test is to identify the bandwidth requirement to support the target users. In that case, if the network is heavily loaded with other traffic, then it’s obvious that amount of data sent to/from client and server would have an impact. Hence, in this case the performance test should be conducted in a separate LAN wherein all the server and test machines (used for client traffic simulation) are located within the same dedicated LAN.

Load Generators

The load testing tools are normally used to create the required load on the server. This can be done using the Load Generators. When the performance test tool is configured to simulate 10 Virtual users, then the tool generates 10 threads or process (depending on the configuration setting of the tool) which sends the client requests to the server as per the recorded think time intervals available in the test script. The system should have enough hardware resources (CPU and Memory) to handle the running threads. The number of virtual users to be spawned from a machine depends on the hardware capability of the machine. If a low configuration desktop is used for load generation, then not more than 15-20 virtual users should be spawned. If a high end server is used for load generation, then about 1000 virtual users could also be spawned as long as enough hardware resources are available in the system. In case of testing of SAP applications, the virtual users that can be spawned from an individual machine becomes very limited.

Most of the load testing tool provides a utility called Load Generator, which can be installed in any other machine other than the one where the test is scheduled. A Load Generator can create the configured number of users load on the machine where the utility is installed and the test script gets auto loaded and run for specified number of times during the test. For example, in order to run a 1000 users test, say load generators can be installed on 5 machines and can be configured to create 200 virtual users load per machine. The user load created on 5 different IPs is sent to the server, as the load generators are configured to run the test scripts that are allocated in the scheduler (controller), from where the test is run. From the test scheduler, the load generated by each agent could be viewed separately or collated (depending upon features supported by the test tool).

Test Data Generators

During the performance tests, most of the time the local test bed might not have the required database records as in the target production database. The target production database volume or projected data volume needs to be studied and accordingly the local test bed needs to be populated with the required number of records. Depending upon the complexity of the test data, it could be done either by the Performance Test Engineer or DBA. Many tools are available in the market in order to generate the test data. Alternatively, database dump taken from the target production system can be used to create the required database records in the local test bed. But in cases where there are security concerns about using the production data in the test environment, depersonalized version of production data can be used if the application is able to handle the depersonalized data without any issues. Conducting the performance test with appropriate database volume is very essential. The query execution time of a SQL could be very less when the database records are around 1000, but when the database volume becomes 10000 records, the SQL execution time could definitely have an impact to fetch all the records and perform the processing as per the business logic. There is a high possibility that some database issues would be hidden if the application is not tested for the actual expected database volume. The way the records are arranged in the table and indexed might create a huge performance impact on the system.

Tips for writing successful performance test scripts

The Test scripts are generated to mimic the user behavior during the performance tests. The Test scripts needs to be modeled in such a way that it exactly simulates the user actions and user think times. The scripts are generated by record and playback feature provided by the test tool and a series of below customizations needs to be done to make the scripts more realistic.

o It is recommended to script or configure (depending upon the feature provided by the tool) in such a way that login and logout transactions are executed only once and rest of the transactions. Incase of Load Runner tool, always place the login transaction inside the vuser_init() and logout transaction inside vuser_end(). This helps to achieve more realistic usage pattern and also helps to close the user session (by enforcing execution of logout transaction) even if the script is stopped in the middle of execution.

o Provide test data variables in the script to enable the script to supply different combinations of the data for different users as in real time.

o Configure appropriate test data usage options (Unique or Random or Sequential).

o Provide appropriate naming convention for each user action to uniquely identify the response time metric for all the transactions of business interest.

o Provide wait times between each of the user action in order to simulate the real time end user behavior. The wait time or think time refers to the user time for thinking or navigation time.

o Handle the unique dynamic values generated by the server i.e. session values, flow id, descriptors, etc to make the script robust enough for simulating multiple users.

o Provide customization in the script navigation logic to handle the user behavior.

o Add logic to verify the correctness of the page responses.

Real time versus Virtual user Mapping

Though Performance testing is conducted for the target number of user load, it’s Performance Test Engineer’s responsibility to assure that each simulated virtual user is equivalent to the real time user, only then the load generated on the server could be realistic. For applications which have the real time usage history (moving to production environment for second time or later), the server load (requests handled per second) during the peak traffic hour needs to be compared with the server load created on the server during the performance test. This helps to confirm the load simulated during performance test is realistic. The think times provided in the test scripts needs to be adjusted so as to create the appropriate load on the server. Analyzing the server load in terms of requests per second is recommended as it is more granular and appropriate unit to measure the load handled by the server rather than the user load measured in number of users. The number of users handled by the server is a user perceived metric which needs to be reported to the business stakeholders, whereas Performance Test Engineer’s focus should be on the number of requests handled by the server.

Chapter 5: Overview of Performance Testing Tools

2008-12-02T18:36:00.003+05:30

Software Performance Testing Handbook
A Comprehensive Guide for Beginners

Open Vs Licensed Performance test tools

There are lot of open source and licensed tools available in the market to do performance testing. Usually for small organizations where there are budget constraints, open source tools would be the only choice. There are good open source performance test tools like JMeter, OpenSTA available in the market. One should be very careful in using custom build or open source tools as the tool implementation logic including thread handling, inter-arrival time between the requests spawned, etc would have a huge impact in the way performance tests are conducted.

When it comes to organizations which have performance critical applications, due to expectation on high accuracy and reliability on the test results, they prefer using licensed tools in the market like HP Load Runner, HP Performance Center, Segue’s SilkPerformer, Empirix E-Test, Radview Webload, Compuware QALoad, etc.

Criteria for choosing the Performance test tool

There are certain things which need to be analyzed before choosing the performance test tool for your application. Look for the following before you choose the tool

1. Look for the protocols supported by the test tool. Does the tool support web applications and client-server applications?
2. Check whether the tool provides the editor to develop the test scripts.
3. Check whether the tool provides features to identify the transactions uniquely, to provide user think time, to provide the test data, to handle server side dynamic variables and to see the script replay with the application pages.
4. Check whether the tool has options to configure the ramp up, ramp down and steady state load duration for running the test.
5. Check whether there are provisions to monitor the system resources during the test.
6. Check whether monitors could be used across different operating systems.
7. Check whether the tool provides the test results in a readable format (which can be opened as html or word format)
8. Check whether the tool provides enough information about the Hits/sec, Throughput (in transactions/sec and bytes/sec), Running users/sec, System resource monitors and errors/sec details.
9. The tool should not consume much of the resources of the client machine resulting in load generation issues.

The most important tool

Choosing the correct automation test tool is not the challenge in conducting the performance testing. The performance test tool could only give the details on the server performance, but it’s the Performance Tester who needs to interpret the observations clearly and analyze the root cause of the issues, confirm the bottlenecks by rerunning the tests, provide the report of observations and recommendations in user understandable format. The Performance tester‘s brain is the most important tool to make the performance testing a success.

The Performance tester should be strong in the concepts rather than just being a performance test tool expert. Though expertise in the test tool is an added advantage, the tester should possess right set of capabilities to pinpoint the performance issues of the application.

Performance Testing Tools Overview

Some of the popular performance test tools include

• HP’s Load Runner / Performance Center (licensed)
• Radview’s Webload (licensed)
• Compuware’s QAload (licensed)
• Borland’s SilkPerformer (licensed)
• Empirix’s e-Load (licensed)
• OpenSTA (freeware)
• JMeter (freeware)
• Grinder (freeware)

Chapter 4: Workload Identification

2008-12-02T18:32:00.003+05:30

Software Performance Testing Handbook

A Comprehensive Guide for Beginners

What is Workload?

The Workload refers to the user load created on the server by the real time user access or during the performance tests. As per G.Kotis words, “a workload can be defined as all set of inputs (programs, commands, etc) from the users to the system from the environment”. For example, for a UNIX terminal, the way how the end user enters the system commands in the terminal is the workload. The workload of a specific system varies from the other.

The workload could be either natural or synthetic. The natural or real workload is the natural realistic actual load in the real time production environment. The synthetic workload mimics the real time behavior of the system and hence it models the natural workload. The characteristics of a synthetic workload could be controlled.

The workload selected for carrying out the tests should closely resemble the real time workload pattern of a web application. If there is more deviation from the test workload and the real time production system workload, then the performance test results will not give accurate results of the system performance in production environment.

Web log Analysis

The web log analysis refers to the analysis of the log files of a web server (Example: IIS or Apache web server) and derive metrics about the access pattern of the real time users. There are various formats used for logging the information in a log file. Some of the ASCII file formats include W3C Extended File format, IIS log file format, NCSA common log file format. Different format uses different time zones to represent the activities. (W3C Extended file format uses Coordinated Universal Time (UTC) same as Greenwich Mean Time (GMT).

The sample W3C Extended File format and IIS log file format are provided below.

#Fields: time c-ip cs-method cs-uri-stem sc-status cs-version

Example:

17:42:15 172.16.255.255 GET /default.htm 200 HTTP/1.0

#Fields: date time c-ip cs-username s-ip s-port cs-method cs-uri-stem cs-uri-query sc-status sc-bytes cs-bytes time-taken cs (User-Agent)

Example:

2005-12-11 00:00:22 131.127.217.67 - 131.127.217.69 80 GET /scripts/mycalender.dll Test 200 282 270 0 Mozilla/4.06+ [en] + (WinNT; +I)

Web Log Analysis Tools Overview

As the web server log files cannot be easily interpreted, analyzer tool needs to be used to analyze the access requests and report statistics. There are many web log analytics tools available in the market to analyze the log file contents and to report statistics graphically. The choice of the web log analysis tool depends on the type of information required and budget constraints.

Some of criteria for selecting the log analysis tool are

Ø What do you need to measure?

Ø Is the interface easy to use?

Ø How fast are reports produced?

Ø Are reports and terminology in plain English?

Ø What is the maximum log file capacity that can be analyzed?

Ø How accurate and reliable are the reports?

Ø How often do you require log analysis reports?

Ø Is it possible to access the tool via web?

Ø Is it possible to open the reports without any thick client?

Ø Is it possible to do customization on the reports?

Ø Can we apply filter on specific reports?

Ø Does the tool provide online help? Is the documentation easily understandable?

Ø Does the tool support a variety of web servers?

Ø What the users of the tool say about the tool? Is any comparative rating available which appreciates the usage of the tool?

Ø Is it possible to export the reports to a MS Word/Excel or PDF format?

Web Log Analysis Metrics

The web server log file analysis provides a lot of information about the visitor behavior which helps in identifying the exact load on the system during different time periods. The historic traffic trends and deviations in expected traffic trends can be identified and compared, which helps in business decisions. A lot of people with varying background like Business owners, Domain managers, Site administrators, Performance Engineers, Capacity Planners, etc, use the log analyzer tools to arrive at server load and for user navigation trend analysis.

The following are the some of the data which could be identified from the web log analysis tool.

Number of Visitors

It represents the count of visitors accessing the site at any point of time. A visitor can make multiple visits.The visitors are identified uniquely through the IP address. By default, a visitor session is terminated when a user is inactive for more than 30 minutes. So a unique visitor (with unique IP address) may visit the web site twice and be reported as two visits. The tool should provide information about the visits and the unique visitors at any point of time.

Number of Hits

It represents the count of requests (hits) for any resource (example: image or html page) on a web server at any point of time. The hits are not equal to the number of pages accessed. For example, if a web page contains 5 images, a visit to this page generates 6 hits on the web server i.e. one hit for the web page and 5 hits for the image files on the html page. Identifying the peak hits per second during the peak rush hour will provide a realistic picture of the peak traffic. But most of the tools do not provide the drill down capability to find the peak hits per second data.

Number of Page Views

It represents the count of requests for the web pages (example: .jsp / .html file) on a web server at any point of time. The page views are equal to the number of pages accessed. If a visitor views 3 pages on a web site, then it generates 3 page views on the web server. Each page view might consist of multiple hits, as the page view includes the images files on an html page.

Authenticated Users Report

It represents the list of users whose user names are authenticated as required by the web site. The information including the number of hits generated by the authenticated users and the region or country from where the authenticated users have accessed the web site is also provided.

Peak Traffic Day à Peak traffic Hour

It represents the daily, weekly, monthly server load statistics and drill down capabilities to find the peak traffic day and peak traffic hour in the entire period under analysis. This is identified by referring the number of visitors accessing the site and the hits generated during the access period. Identifying the peak traffic day and peak traffic hour helps to know the maximum traffic supported by the web site.

Throughput / Bytes Transferred

It represents the total number of bytes transferred by the server to the client(s) at any point of time. It is an important metric which portrays the server performance at any point of time. The peak throughput metric helps to know the maximum data that is transferred by the server to resolve the client requests during the peak hours.

Files Downloaded Report

It represents the popularity list of all the files including the web pages, images, media files, etc downloaded by the users. Files are ranked by the number of times they are requested by the visitors (number of hits). This report includes files with all extensions.

Top Downloads Report

It represents the popular files downloaded from the web site with the extensions .zip, .exe, .tar, etc. It does not include the image files, html pages, etc. It also provides the bytes transferred information of how many total bytes of data were transferred by the web server to the visitors for each downloaded file.

HTTP Errors

It represents the HTTP response errors sent by the server during the access period. The summary of HTTP error codes and the time of occurrence is a useful metric to identify the server behavior. The typical errors reported are page not found errors, incomplete download errors, server errors, etc. The following are the five classes of response error codes.

1XX (Informational) : Request received, continuing process.

2XX (Success) : The action was successfully received, understood and accepted.

3XX (Redirection) : The client must take additional action to complete the request.

4XX (Client error) : The request contains bad syntax or cannot be fulfilled.

5XX (Server error) : The server failed to fulfill an apparently valid request.

Most significant Entry & Exit Pages

It represents the top entry and exit pages used by the users to login/logout to/from the web site. Some web sites have more than one option to login (For example: through registered user’s login page, by Google search, by download page, etc) or to logout in different ways. The predominantly used user option can be identified by using this metric.

Browsers

It represents the information about the web browsers used by the users to connect to the server. The set of various browsers used, the user agents and the percentage usage by the user are provided.

Platforms

It represents the information about the operating system used by the users to connect to the server. The set of various platforms used and the usage percentage are provided.

Geographical regions à Top Countries à Top Cities

It represents the top geographical regions, top countries and top cities from which the users accessed the server. It provides the IP details, bytes transferred, hits generated, etc per region, country and city breakup.

Visitor Path Analysis

The visitor arrival rate can be used to identify the statistical distribution pattern (Self Similar distribution, Poisson distribution, Exponential distribution, etc) of the application. This metric is usually not provided by the log analysis tool. This is a user derived metric which needs to be calculated by the performance tester based on the mean arrival rate of the request and the peak server load details.

Visitor average session length

It represents the average visitor session duration in minutes. The session length of visitor groups is provided in a sorted order.

Search Engines and Robots

The Search engines are the information retrieval systems which allow the users to search for specific web sites based on the keyword search. The popular search engines are Google, Yahoo, MSN, ASK.com etc.

A web crawler (also known as a Web spider or Web robot) is a program or automated script which browses WWW in a methodical, automated manner. It is an automated Web browser which follows every link it sees.

The Log Analyzer provides the list of search engine's name and key phrase that referred visitor to the website. The Top Search Engines Report is a list of search engines used by the visitors to find out the website, ranked by the number of referrals from each search engine. The Referring Sites Report shows the referrer websites that drive visitors to your site ranked by the number of hits received from that referrer. The Spider History Report illustrates day-by-day history of search spiders visits.

Overview of Statistical distributions

The Performance Test Engineer needs to identify the underlying statistical distribution of the application’s user request arrival pattern. This helps to extrapolate the server load (requests handled per unit time) for high user load. For example, if two applications (A1 and A2) have the similar target load objective (say 1000 users) but different statistical distribution (say A1 – Poisson and A2 – Self similar), then the applications A1 and A2 needs to be performance tested for different maximum server loads in order to certify the applications for 99.9% availability. The maximum load point differs based on the type of the distribution which is very vital to identify the system performance for that maximum load point. The statistical distribution pattern analysis helps in deciding appropriate load requirement during performance tests.

The probability distributions are a fundamental concept in statistics. Discrete probability functions are referred to as probability mass functions and continuous probability functions are referred to as probability density functions. The term probability functions cover both discrete and continuous distributions. When we are referring to probability functions in generic terms, we may use the term probability density functions (PDFs) to mean both discrete and continuous probability functions.

Chapter 3: Deriving Performance Test Goals

2008-12-02T18:24:00.002+05:30

Software Performance Testing Handbook
A Comprehensive Guide for Beginners

Why to set the Performance Test Goals

It is always required to decide what we are trying to achieve and then take quality steps in an organized way to achieve it. The performance test goals are important to check whether the system meets its expectations. In order to claim that the system performance is good or bad, one need performance test goals to be in place to justify the claim quantitatively.

The Performance Test Engineer needs to certify the application for performance based on the goals set by the stakeholders / derived with the help of stake holder’s inputs.

Know your customer

How to get the performance requirements from non-technical customers is a valuable question that everyone has. In most cases, customer expectations are either incompletely defined or more conceptually defined. Customers are not expected to be the performance test experts. (If so, then we are jobless!). We need to better understand their expectations by asking right set of questions and translating their conceptual requirements into more quantitative goals. The performance tester needs to play the role of a modem translating between the novice user language and performance testing terminologies. It is the performance tester’s responsibility to get the essence from the customer.

Talk in Customer’s Language: One needs to quickly understand the customer and should start talking in their language. Make a note that you don’t simply ask questions like what is the Hits/sec goal, response times expected out of various transactions and make the customer to feel uncomfortable. In one of my projects, when I asked the above set of questions, I got an unexpected answer from them, ‘Hey, in fact we also wanted to know answers for these questions. That’s why we have planned for the Performance Testing’. Though their answer surprised me; then I slowly understood that I am not talking in their language.

Look for Customer’s Expectation: Some customers may be clear in their objective; some may have conceptual expectations on the performance test results. Try to understand their language and thereby their expectation. Don’t talk using performance testing jargon and make the customer feel that they are ignorant, which is not the case. Don’t expect that your customer will give you the performance goals in a single shot. Rather , start the discussion with an overview of important terminologies like what response time actually means, what is a Hit , what is a transaction , what is a performance goal , why is it required before starting the performance tests, etc so that the customer feels comfortable to explain what he/she wants. Try to educate the customer and explain with an example to understand what they are expecting from the system.

Gain Customer’s Confidence: Try to build a good rapport with your customer. Try to make them to realize that you care for them. Try to briefly explain about how you have solved the issues in some other projects in your past. Try to make them feel that you have achieved great things in your past projects. That’s the first impression you get from them. Once you get that, you will have the full liberty to state your expectations from them in order to provide successful results. Don’t scare the customer talking too much technical rather make them to understand the results in a very polite way. More than results, your way of expressing the result and your communication style impresses the customer.

Performance Test Estimation

Performance Testing forms an investigation activity and hence it is very difficult to predict the efforts required for performance testing. But by experience I have come across various factors which impact the effort requirement for all the test activities during performance testing. As a best practice, every performance test project should get started with the project kick off meeting where the high level performance test objectives and application overview is discussed. The discussion inputs need to be considered for the effort estimation plan. The effort estimation needs to be approved by the stakeholders before continuing with the performance test plan development activity.

The performance test effort requirement depends on the performance test scope of the project. The effort required for each life cycle phase of performance testing depends on the project category. The broad classification of the performance test projects are as follows:

• Verification / ReTest: Mostly this category projects will be simple in nature and it requires a recheck on the application performance due to minor changes in the application / infrastructure.
• Application Benchmarking: Mostly this category projects will be of simple to medium complexity and it requires certification of the application for the supported load levels without any detailed bottleneck analysis or tuning activity. In most of the cases, the application is tested for the first time.
• Regression Test: Mostly this category projects will be of medium to complex category and it requires redoing of the entire performance testing for the application which was performance certified earlier due to major functional changes or fixes.
• Performance Testing: Mostly this category project will be of medium to complex category projects and it requires complete effort for all the test phases and it includes bottleneck analysis and tuning based on the project requirements. In most of the cases, the application is tested for the first time.

Chapter 2: Performance Testing – A Quick Overview

2008-12-01T21:03:00.004+05:30

Software Performance Testing Handbook

A Comprehensive Guide for Beginners

Performance Management

Reactive Approach
The Performance testing activity is often considered as a reactive way of performance management. In most of the cases, the system performance is never thought of during the early phases of Software Development Life Cycle phases (SDLC). Performance Testing is often thought of only as a last activity after the system testing phase.

Also, if the performance bottlenecks are related to the system architecture or the system design, then it becomes highly impossible to fix the issues due to the high COPQ (Cost of Poor Quality) and in certain cases, the system is put into trash because of the huge deviations in the performance constraints.

Basically waiting for the performance problems to appear and then dealing with it at the end is always not a better approach. Hence performance testing is considered as a reactive approach as there is not much importance given to the system during early life cycle phases. It is more a ‘fix-it-later’ approach which is not that effective.

Proactive Approach
The Proactive approach anticipates the performance problems well in advance and adopts techniques to mitigate them. The importance of performance is thought about during all the SDLC phases right from the requirement analysis phase and various performance engineering activities are identified for the system.

The disadvantages of ‘fix-it-later’ approach are well understood and engineering practices are adopted to analyze the system design in performance angle. As the system is evaluated for the performance right from the design phase, the chances of last minute surprises is very less in proactive approach.

Application Performance Management

Application Performance Management (APM) is about managing the performance of the application throughout its lifecycle to ensure availability and better performance to the end users of the system. It forms a closed loop of performance management by managing the performance in development, testing and production stages of the application. Lot of production monitoring and diagnostics tools are available in the market to identify the performance problems of the production system and to provide quick diagnostics to resolve the performance problems. It also provides lot of inputs to carry out the pre-production performance testing. Some tools provide flexibility to use the same test assets (scripts, etc) for both pre-production and post-production performance testing.

Myths about Performance Testing

Organizations generally like to make the headlines with their success stories and not about the failure web sites due to unexpected user load. Industry study reveals that poor performing IT applications cost industrialized nations almost $45 billion annually. Most of the organizations understand the importance of performance testing and the impact of bypassing it. ‘How much time would it take to do the performance testing of an application?’ is a million dollar question for all organizations wherein the Software Performance Engineering (SPE) is not in place from the start of the project. Most of the organizations plan for the performance testing of the applications towards the end of the project, though it is of great importance to decide whether the performance bottlenecks are addressed before the go-alive dates in order to meet the end user demand on the production systems.

The following are the some of the popular myths about Performance Testing.

MYTH 1 : Performance testing is the last activity thought of only based time availability.
FACT : The importance to the system performance should be thought from the requirements identification phase and performance engineering activities needs to be practiced during each phase of SDLC.

MYTH 2 : Conducting Performance testing would increase the system performance irrespective of implementing the recommendations.
FACT : Conducting Performance test alone will not improve the system performance. It helps to identify whether the system meets the performance test goals and identify the performance bottlenecks of the system.

MYTH 3 : Performance Testing is just doing code profiling or memory profiling to tune the code.
FACT : Performance Testing is about evaluating the system for its conformance to the performance test goals and thereby identifying the performance bottlenecks in the software and hardware of the system.

MYTH 4 : Performance Testing needs to be done on all the functional flows of the application to identify performance issues.
FACT : Performance Testing needs to be done for specific scenarios based on Pareto analysis. The scenarios that are used often, which are of stakeholder’s concern, high critical scenarios, scenarios which are considered error prone are the right candidate for conducting performance testing.

MYTH 5 : The response time goal should be per the industry standards.
FACT : There is no industry standard for response time. The response time goal needs to be derived based on the hardware and software capacity available for the system considering the end users tolerance limit.

MYTH 6 : Instead of investing in performance testing activities, it is better to procure high capacity hardware as it is cheap.
FACT : Industry studies show that the hardware price is improving at around 40% per annum whereas the demand for IT resources is increasing at around 60% per annum. The reason for the actual performance issue needs to be identified through performance testing activity.

MYTH 7 : Performance Testing can be planned in parallel during the functional testing.
FACT : Performance Testing needs to be planned on a stable system only after the completion of system testing phase.

MYTH 8 : An application needs performance testing once in its life time irrespective of how many modules of the application are revamped over a period of time.
FACT : Performance testing needs to be best done as a continuous process of measuring, analyzing and tuning the system performance. The system performance needs to be re validated whenever the code is changed or added in the system or if there is a hardware change.

MYTH 9 : The Performance bottlenecks can be identified by conducting one test run.
FACT : The Performance bottlenecks cannot be identified by conducting one round of test. The isolation of bottlenecks becomes very tough in complex systems. Multiple rounds of isolation tests need to be run in order to identify the performance bottlenecks depending upon the type of bottleneck.

MYTH 10 : Performance Testing can be done only if the system is properly tuned.
FACT : Performance Testing can be done on any system to identify the bottlenecks. Irrespective of the system condition, performance testing can be conducted.

MYTH 11 : The Performance Tester is solely responsible for detecting and tuning the system for performance.
FACT : It is often a joint effort to detect the performance bottleneck and tune the system to meet its performance goals. Advices from Subject matter experts, DBAs and System administrators become a value add to identify/isolate the bottleneck easily.

MYTH 12 : Performance Testing can be carried out only on the development / test environment.
FACT : Performance Testing needs to be planned on an isolated environment in order to isolate the performance bottlenecks easily.

Performance Process Maturity Model

Michael Maddox proposes five levels of the maturity model for the performance test process. All the levels have the following common characteristics:

1. The levels are cumulative. The performance activities and processes practiced at level 2 are retained and enhanced at level 3 and so on through higher levels.
2. Different applications may exhibit different maturity levels.
3. Some level of learning and feedback is applied as work progresses. Organizations at higher levels of maturity apply more effective, more strategic feedback.

Chapter 1: Introduction

2008-12-01T20:49:00.007+05:30

Software Performance Testing Handbook

A Comprehensive Guide for Beginners

Why Performance Testing is Crucial

Performance Testing is an essential activity for developing optimal software solutions. It is highly important to avoid last minute surprises that arise after deploying the application in the production environment. The failures experienced by the end users are vulnerable to the competitors.

Performance Testing helps to answer the sample of following questions:

• What is the response time of system during the expected load condition?
• How is my system behaving during unexpected load condition?
• Is my system scalable to specific user load?
• Which environment provides the best performance for my system?
• Is my system performing well after adding the new NIC card?
• Will my system handle the spike user loads? Or will it crash?
• Does my system need a dedicated database server to meet the performance goals?

The performance being the essential quality of software application, it is required to evaluate the software systems for its performance.

Performance Testing Jargons

The following are the some of the main performance testing terminologies used in day today life. A good understanding of the below jargons are critical to get the most out of this book.

Business Transaction: It refers to a sequence of request-response and the related information exchange between the client and the server in order to accomplish a business requirement. For example, an online banking customer transferring money from one account to other is a business transaction.

Test Scenario: It refers to a high critical business transaction or high traffic workflow identified for the performance testing. Normally all the business transactions cannot be tested for its performance. The Paretto analysis helps to identify those specific 20% transactions which is often exercised by the end users rather than testing the remaining 80% non-critical transactions.

Think Time: It refers to the time taken by the user for thinking or clicking on any of the web page links, buttons, etc while navigating through the web site. Think time is a very important parameter which needs to be set while scripting the scenarios using the performance testing tools. The business analysts of the application or web site management team or sometimes even the end user survey might give a realistic picture about the think time requirements of a transaction.

Virtual User: The Performance testing tool simulates the real user traffic by creating virtual users during the performance test. A virtual user is configured in the performance test tool to run the script simulating the real user behavior. Though in real time, users access the application from a unique IP address, more than one virtual user can be configured to access the application from the same IP address.

Simultaneous User load: The simultaneous users have the active session on the server at any point of time wherein each user will be executing different transactions. For example, if we say 100 simultaneous users load, then there will be 100 active sessions opened up in the server, wherein each user will be performing different set of transactions – one logging in, another viewing reports, another navigating to the next page, etc. The simultaneous user load of a web site would be always greater than the concurrent user load of a web site.

Concurrent User load: The concurrent users connect to the server and perform the same operation at any point of time. For example, if we say 100 concurrent user load, all 100 users would be logging in at the same point of time, view the reports at the same point of time, etc. For example, an online banking web site might have 10,000 – 20,000 simultaneous user load, but 1000 to 1500 concurrent user load.

Performance Requirements/Goals: It refers to the quantitative way of putting forth the criteria for claiming the system performance is good. The performance requirements of an application are often expressed in response time, hits per second, transactions per second, etc. The performance requirements of the System Under Test (SUT) need to be quantitatively expressed in the performance test plan.

Workload: It refers to the user load subjected by a web application under real time user access or during the performance test and the way the users are distributed between various transaction flows. Normally, the web server log files of a web site can be analyzed to learn about the workload of the site (if the web site is already in production). For web sites which are yet to go alive for the first time, a workload model needs to be developed based on the discussion with business analysts, application experts, etc. It is very important to know the workload of the application before conducting the performance test. Conducting the performance test for a system without proper analysis on the workload might lead to misleading results.

User Abandonment: It refers to the situation wherein the end users exit from the web site because of the slow performance of the site. The user abandonment rate varies from web site to web site. A low priority web site might experience a high user abandonment rate compared to the payroll web site. Analysis on the abandonment rate of the site needs to be done before concluding on the load on the site.

Baseline Test: It refers to the test conducted to measure the application performance for 1 virtual user load. The baseline test is often conducted to collect the metrics about the system performance for 1 user load and thereby it could be used for comparing the system performance at a high load condition. It also helps to check the validity of the test script.

Benchmark Test: It refers to the test conducted to measure the application performance for a low load condition. Normally for a benchmark test, 15 -20% of the target load can be considered. The objective of the benchmark test is to validate the correctness of the test scripts and to check the readiness of the system before subjecting it to a high load. It helps to know whether various components of the system collaborate as per the design and meet the performance test SLA for 20% of target load.

Hit: It refers to the server request for accessing a web page or a file or an image from a web server. For example, if a web page contains 5 images, then a user visit to that page creates 6 hits on the web server (5 hits for fetching each of the images and 1 hit for fetching the web page). For a web site, the performance requirement can be derived in terms of hits per unit time like ‘the system should support the load of 10 hits per second with a response time of below 5 seconds’.

Response Time: It refers to the servicing time or processing time in order to serve the request. It is measured from the time the web browser sends the request to the web server to the time when the first byte of response is received by the requesting user after server processing. The response time of a transaction say login refers to the time taken by the server to respond to the request by logging into the system and displaying the home page (it includes web server processing time + application server processing time + database server processing time + network latency).

Throughput: It refers to the amount of data (in bytes) transferred by the server in order to the serve the client requests. It is a good indicator of server performance as it refers to the amount of work done by the server. Throughput also referrers to the number of requests or transactions processed by the server at any point of time. For example, the server throughput can be expressed as 2.5Mbps or 35 Hits/sec or 8 Transactions/sec.

Page Views: It refers to the number of pages transferred by the server in order to serve the client requests. i.e. number of times the web site is completely loaded or refreshed. A page is a html page or script page or plain text documents. It is also a good indicator of server performance. If a user visits a web site and navigates to 5 pages and then logs off then the Page Views could be indicated as 5.

Performance Bottleneck: It refers to the slow spot, the effects of which are widely felt. It refers to the situation/areas which do not allow the application to perform as per its ideal specifications. For example, the response time increase for the load of 100 virtual users because of improper setting of HTTP connections parameter in the IIS server, CPU utilization reaching 95% during 100 users load are typical performance bottlenecks. A bottleneck might lead to a failure if mitigation actions are not taken.

Load Test: It refers to the test conducted on the system simulating the actual usage pattern or a representative load in order to identify the system behavior and diagnose the performance bottlenecks in the system. The load test should have the objective of checking how the system performs for the specific load, say 1000 simultaneous users. It is carried out by ramping up the user load from zero to the maximum count with the specific ramp up rate depending upon the kind of application and the user access pattern. Sometimes load testing is conducted to identify the maximum acceptable user load or the highest arrival rate the system can handle.

Stress Test: It refers to the test conducted by subjecting the system to an unreasonable load to identify the system breakpoint. It is generally considered as a negative testing as the system is subjected to an unrealistic load. It is usually conducted to know the importance of potential harm created during unrealistic load and to act on it proactively. It also sometime helps the web site owners to inform the user base about the scalability limit of the application.

Spike Test: It refers to test conducted by subjecting the system to a short burst of concurrent load to the system. This test might be essential while conducting performance tests on an auction site wherein a sudden load is expected.

Volume Test: It refers to the tests designed to measure the throughput of the system more in a batch processing, messaging kind of environment. The objective of this test is to identify the processing capacity. Also, in database server perspective , a volume test can be conducted to test the server performance by subjecting it to huge data (say , 75,000 rows of employee information in EMPLOYEE table) keeping in mind the data available by end of 2 years of web site usage.

Stability / Longevity / Endurance Test: It refers to the tests conducted to check whether the system is able to withstand the load continuously for a longer duration say 48 hours, etc. The system is subjected to reasonable representative load and tested for its continued performance. These tests are helpful in identifying memory problems (memory leaks, buffer overflow, etc) which arise after continuous usage of the application for a longer duration.

Bottleneck Isolation Test: It refers to the specific tests conducted on the system or a specific component of a system to isolate or confirm the performance problem. It is often conducted during or after the load tests to identify the root cause / confirm a specific performance issue. Mostly these tests are decided dynamically to debug the performance problems.

Software Performance Testing Handbook - Preview Index

2008-12-01T20:38:00.014+05:30

Preview available for quick reference of audience

Software Performance Testing Handbook
A Comprehensive Guide for Beginners

Chapter 1:Introduction - Preview

Chapter 2: Performance Testing – A Quick Overview - Preview

Chapter 3:Deriving Performance Test Goals - Preview

Chapter 4: Workload Identification - Preview

Chapter 5: Overview of Performance Testing Tools - Preview

Chapter 6: Setting up Performance Test Environment and Performance Test Scripts development best practices - Preview

Chapter 7: Application Benchmarking - Preview

Chapter 8: Performance Test Execution - Preview

Chapter 9: Performance Test Monitoring - Preview

Chapter 10: Performance Bottleneck Analysis - Preview

Chapter 11: Performance Test Reporting - Preview

Chapter 12: Road Ahead – Moving towards advanced Performance Engineering techniques - Preview

My Book on Software Performance Testing for Begineers

2008-11-27T20:05:00.002+05:30

Here is the updated table of contents of my book.... I am happy to share that the book will be available in my blog in downdable pdf format within a week. Preview of all the chapters will be available in couple of days.

Software Performance Testing Handbook

A Comprehensive Guide for Beginners

Author : Ramya Ramalinga Moorthy

Contents

Chapter 1: Introduction.. 7
What is Performance Testing.. 7
How Performance Testing is different from Functional Testing.. 8
What is an E-Business Application.. 8
Components of a Web Site.. 9
Why Performance Testing is Crucial.. 10
Performance Testing Jargons.. 10

Chapter 2: Performance Testing – A Quick Overview.. 15
Performance Management.. 15
Performance Testing Approach.. 16
Dynamism in Performance Testing.. 18
Application Performance Management.. 19
Myths about Performance Testing.. 19
Performance Process Maturity Model.. 22

Chapter 3: Deriving Performance Test Goals.. 25
Why to set the Performance Test Goals.. 25
Know your customer.. 25
Deriving Quantitative Goals.. 26
Industrial Standards for Benchmarks.. 27
User behavior Modeling.. 27
Importance of automation in Performance Testing.. 29
Performance Test Estimation.. 29

Chapter 4: Workload Identification.. 33
What is Workload? 33
Types of Workload.. 33
Customer Behavior Modeling.. 36
Customer Behavior Model Graph (CBMG).. 36
User Community Modeling Language (UCML).. 36
Web log Analysis.. 36
Web Log Analysis Tools Overview.. 37
Web Log Analyzers Summary.. 38
Web Log Analysis Metrics.. 42
Web Site Traffic Monitoring.. 46
Overview of Statistical distributions.. 46

Chapter 5: Overview of Performance Testing Tools.. 51
Performance Test Tool Requirements.. 51
Market Tools.. 52
Open Vs Licensed Performance test tools.. 52
Criteria for choosing the Performance test tool.. 52
The most important tool.. 53
Performance Testing Tools Overview.. 53

Chapter 6: Setting up Performance Test Environment and Performance Test Scripts development best practices.. 56
Know the Test and Production environment.. 56
Test environment Isolation.. 56
Network Isolation.. 57
Load Generators.. 57
Test Data Generators.. 58
Test Execution Environment.. 58
How to choose the test scenarios.. 59
Tips for writing successful performance test scripts.. 60
Real time versus Virtual user Mapping.. 63

Chapter 7: Application Benchmarking.. 64
What is benchmarking?.. 64
Why should we benchmark the applications?.. 64
Software Benchmarking.. 64
Hardware Benchmarking.. 65
Industry Standards for Benchmarking.. 65
Transaction processing Performance Council (TPC).. 65
Standard Performance Evaluation Council.. 66
Mistakes in Benchmarking.. 67

Chapter 8: Performance Test Execution 68
Quality of Service of web application.. 68
Virtual User Simulation.. 69
Pre-requisites for Performance Tests.. 69
Types of Performance Tests.. 70
Performance Test Execution Approach.. 73
Tips for Load and Stress Test Execution.. 76
Test Execution Factors.. 77

Chapter 9: Performance Test Monitoring 81Introduction to Performance Monitoring.. 81
What Counters needs to be monitored?.. 81
Key Performance Counters.. 82
Performance Monitoring in Windows platform.. 85
Using Perfmon.. 85
Post production Monitoring.. 87
Benefits of Performance Monitoring.. 87

Chapter 10: Performance Bottleneck Analysis.. 88
Scott’s Recommendation on Scatter Charts.. 88
Scatter Chart Types.. 89
Performance Test Analysis.. 90
Best Practices on Performance Bottleneck Analysis and Isolation.. 92
Performance Testing Vs Profiling.. 94
HP Diagnostics.. 94
SQL Profiler.. 98
Know your Limits.. 99
Typical bottlenecks on DB server.. 100

Chapter 11: Performance Test Reporting.. 102
Why is it Critical.. 102
How to report the results of performance testing.. 102
Components of a good Test Report.. 103
Components of a Good Visual 104
Response Time Reporting 104
Best Practices of Test Reporting recommended by Scott Barber 105

Chapter 12: Road Ahead – Moving towards advanced Performance Engineering techniques.. 107
Moving towards Performance Engineering.. 107
What is Queuing Theory?.. 107
Types of Queuing System.. 108
What can be modeled as Queues?.. 109
Limitations of Queuing Theory.. 110
Operational Laws.. 110
a. Utilization Law.. 110
b. Little’s Law.. 111
c. Interactive Response Time Law.. 112
d. Forced Flow Law.. 112
e. Flow Balance Assumption.. 113
f. Bounding Law of Service Demand.. 113
Capacity Planning.. 114
Why Capacity Planning is required.. 114
When is it required?.. 115

Happy Learning!!!!

Performance Simulation using Ptolemy Framework made easy by PEA

2008-05-09T01:46:00.004+05:30

The moment we start talking about Performance testing, popular market tools, type of tests, bottlenecks and the isolation techniques ,etc comes to our mind. But am sure you would be definetly surpraised to see a group who always talks about simulation , performance modeling , prediction, etc. Yes, they are none other than my favourite reference site of PEA.
Checkout this new tool available in PEA site http://www.pea-online.com/resources.htm. But be prepared for getting lost into the world of new terminolgies and jargons which might scare you little. But try understanding at a high level what is the tool all about and it be a good start to learn more about it.
Even i am very new to simulation and am a begineer to this new world of simulation. But its really interesting and and with lot of excitement am writting this note. Let me give a quick intro of all i understand.
Ptolemy is a open source project developed by group of researchers at U.C. Berkeley. It is a framework for assembling software components, a modeling and simulation tool, a block-diagram editor, a system-level rapid prototyping application, a toolkit supporting research in component-based design, a toolkit for building Java applications,etc. You can correlate it to Rational Rose (or any other modeling tool) used to create UML diagrams which consits of actors and the necessary association and aggregations. In case of Ptolemy, the models are stored in a format called MoML (Modelling Markup Language) which is created by using Vergil, Ptolemy's editor. The MoML is a modelling markup schema in XML. It is the primary mechanism for constructing models using Ptolemy framework.
In Ptolemy,it is easy to create simulation model by dragging and dropping the available actors. It provides set of actors libraries where each actor is a pre-built java class written in the ptolemy framework. You can image an actor as a web server, another actor as a DB server and interconnect them.But there will be lot of input and output parameters which needs to be defined by the user to create a successful model and it is stored as an xml file internally.
Now, it becomes too complex to drag and drop all the required components to simulate the IT system infrastrucutre. PEA has come up with composite actors which are built by using a set of available actors provided by Ptolemy. So that, we can input the required details of the server infrastructure (like, how many servers, tiers, arrival rate and service demand of various service centers,etc) through a web UI and a xml schema can be created from their tool. This will be help the user not to spend time in learning how to use Ptolemy and how to manualy feed the input to each of the actors in the model using Vergil editor. The xml file generated from the PEA tool can be fed into the Ptolemy editor Vergil to generate the text output of the simulation model. The output will be the response time predicted on the simulated server infrastructure.
In nutshell, this PEA tool takes as input deployment configuration, hardware details, workload details, etc, and generates an XML file. This XML file when run using Ptolemy (Open source, Free for use, Berkeley Lab’s Simulation Framework) can be used for simulating the performance of an IT environment. Advantage I see is the it is not required to know about simulation or its in-build actors/features to use it.
Also checkout other free tools available in PEA site. Though those analytical tools are really valuable ones, the real challenge lies in understanding how to use those tool for our system under test practically and interpret the results. Ofcourse, thats their business and formally we might need to approach them and get trained in them to use them for projects effectively.
Happy Learning!!!

2008-04-08T21:36:00.000+05:30

Useful Info about Oracle STATSPACK Analysis

From Perfmon (Windows platform) / SAR , vmstat & other commands (UNIX platforms), you can get the insight of the server resource utilization information. But in case of the database servers, it’s very important that you need to know what is happening in the database server.

Oracle (versions later than 8i) comes with a performance diagnosis tool called STATSPACK. (In case of SQL server, try using SQL Profiler tool).

Statspack is an Oracle performance diagnosis tool (perfstat) available from Oracle 8i. It has the features of Oracle’s utlbstat.sql and utlestat.sql utilities (used in previous versions of Oracle). It has about 25 tables which help to maintain the performance information of the server.

There are good urls available in the net which details the working of Statspack. Some of them include

http://www.pafumi.net/Statspack_Analysis.html
http://www.dba-oracle.com/art_statspack.htm
http://www.akadia.com/services/ora_statspack_survival_guide.html
http://jonathanlewis.wordpress.com/2006/11/29/analysing-statspack-pt1/

There are certain things which a performance tester needs to know about the database server. I feel it’s very much required that a performance tester knows what the various terminologies (atleast) though DBA’s knowledge is not possible. From my experience, I strongly feel that Statspack analysis helps in figuring out whether something is really going wrong in the database. It would help the performance tester to answer the question - Is it worth to spend some time in tuning my database to improve system performance?

At a high-level, it works like this. Before running the test, take a snapshot (by running a simple command) of the database (snapshot collects the current value of all possible database statistics and stores it in the database table. After running the test, take another snapshot of the database and now compare these 2 snapshots (by running a simple command) and get the text report. By analysing this report, you can identify how the database was performing between these 2 snapshots (i.e... during the test run, which is what we are interested in looking at).

I am sure that if we are aware of following terms, it will be easy to figure out certain major database issues. Ofcourse, Statspack analysis performed by DBA is recommended, this is not practically achieved in many projects. In such cases, being a performance tester, you can atleast visualize bottlenecks in the database without DBA’s help.

1. Parses (hard parses vs. soft parses)
2. Transactions per second
3. User calls
4. % Library Hit & % Buffer Hit
5. Top Wait Events
6. Specific Wait events related to Indexing issues/ high table scans
7. Rollbacks per transaction
8. Top 10 SQLs sorted by Query executions, Buffer gets (CPU time) and Parse Calls.

There are lot of Statspack analysis tools available which will analyse the Statspack report and provide the analysis report. But you need to atleast know the above stated information in order to understand the report better. Try using online tools like oraperf.com or statspackanalyzer.com. If you are looking at deep insights of the database server to provide recommendations to the development team, try using tools like spAnalyzer, Wise, etc. (http://www.spviewer.com/index.html , http://www.wise-oracle.com )

Do think about Think Time

2008-03-25T21:24:00.000+05:30

I got a chance to interact with lot of performance testers in various organizations , not only in India, but also in abroad & found that most of us are interested in spending time for setting realistic performance test goals & identifying the frequently used business scenarios (business flows) , but very few analyze the think time of various business flows.

It becomes very important that the average think time of the end users of the application needs to be analyzed rather than estimating specific time limit by counting the total fields to be filled by the end user. For the same application(lets take Yahoo new account creation), a new user might take 4-5 minutes or more to fill the required details to create a new account. But at the same time, a frequent old user, might take 1-2 minutes to create a new account. These think times (4-5 minutes for new user & 1-2 minutes for old user) might impact the server load.

Lets assume 100 new users are accessing the yahoo application for the time interval 13.00 to 14.00 hours & 100 old users are accessing the application for the time interval 14.00 to 15.00 hours continuously. Though there are 100 users (100 active sessions created) load on the server during both 13.00 to 14.00 hours and during 14.00 to 15.00 hours, the actual server load(requests) is going to be different during these 2 hours.

When you have a close look at the server load in requests/sec (Hits/sec) , you will find that the server load during 13.00 to 14.00 hours might be lesser than 14.00 to 15.00 hours, because of high think time used by the new users. If you look at the quantified numbers, you will be shocked to know that during the first hour, the average load could be apprx 12 hits/sec whereas during the second hour, the server load could be around 25 hits/sec.

Having performance test goals in mere user load alone doesnt make the test more realistic. Detailed analysis on the end user's think time makes it more realistic.

Response Time Vs Queue Length Vs Server Utilization

2008-03-25T20:16:00.000+05:30

1. Is the system Response time directly / inversely proportional to the server utilization ?
2. Will there be queues(for CPU/Memory/Disk) if Utilization is less than 100% ?

Actually speaking, there is no correlation between system response time and the server utilization. The Response time increase might happen even when the server is less utililised. (To illustrate this, look at the ATM example available in the PEA site - http://www.pea-online.com/resources.htm).

Most of us dont understand this relationship. Increase in response time (during high load) is caused only because of queueing of the requests. The user request arrival pattern contributes to the high response time. Due to the adhoc user request arrival pattern, long queues are formed in various service centers which leads to high response time.

Response Time - What are the various components of Response time?

Waiting time (Queuing time) + Processing time = Response Time.

For a server, the processing time is always the same irrespective of load. For example, the server might take 2 seconds of processing during 1 user load and during 1000 users load for a transaction. But the waiting time would be high during 1000 users load, which drastically increases the overall response time. So, response time increases, if the waiting time increases which is caused by long queue.

Also, dont think that there will be queues only if Utilization is more than 100%. There can be queues formed even when the Utilization is less. So, everything boils down to knowing the user arrival pattern of the system. This is going to determine, when the system is going to be loaded and break.

Hence, knowing the user arrival pattern of a system is very important rather than setting mere goals to perform load/stress test. Analysing the user arrival pattern of an application would help in setting realistic goals which becomes more of organization logistics issue and most organizations bypass and pay for it later.

Tips to define the Performance Test Strategy

2008-02-25T20:41:00.001+05:30

Best practices in defining performance test strategy

Most of the application would need Load tests, Stress tests and Stability(Endurance) tests to be planned before moving it to production. For data voluminous applications, there might be more emphasis on Volume testing and some applications like Auction site might emphasis on conducting spike tests with sudden spikes. For all other applications, its common that atleast 3 cycles of load tests followed by stress test and endurance test run (atleast for 12 hours) needs to be planned. (This is on a general case, but based on the application context, definetly there will be changes done to this approach).

For any application, irrepective of time availability , its always recommended to start with Baseline test. Baseline test is nothing but one user test which is run to identify the correctness of the test scripts and also it helps is checking whether the application meets the SLAs (Service Level Aggrements) for 1 user load. This values can be used a benchmark for newer versions to compare ths performance improvement.

Next comes the Benchmark test for atleast 15-20% of the target load. It helps in identifying the correctness of the test scripts and tests the readiness of the system before running target load tests.

Its a very good practice to always start with Baseline & Benchmrk tests on any application before conducting the scheduled load tests. I would say, infact its scott barber's way ofcourse.

When it comes to the scheduled tests - Load Tests, always plan to run atleast 3 roundsof load test. Irrespective of doing a slow ramp , its advised to have 3 individual load tests for 50%, 75% and 100% target load. (The load level should be defined based on the system performance rather than just 50%, 75% & 100% target load levels).

Test Scenario - Have a slow ramp up followed by a stable period for atleast an hour and ramp down. During this stable period, the target user load needs to perform various operations on the system with realistic think times. All the metrics measured should correspond only to the stable period and not during ramp up/ramp down period. Dont conclude any transaction response time just based on 1 or 2 iterations. The server should be monitored atleast for a minimum of 5 iterations (at the same load level), before concluding the response time metrics, because there could be some reason for high /less response time at any single point of time. Thats the reason watch the server performance for atleast 5 iterations at the same load level(during stable load period) and use the 90th percentile response time to report the response time metrics.

Test Metrics - Look for 90th percentile response time and standard deviation, ofcourse along with hits/sec and other server resource monitors. More the deviation(more than 1), more bursty is the graph and hence its recommended to rereun the test.

Load Tests should be always followed by Stress Tests - Based on the Load test results, slowly increase the server load step by step and find out the server break point. For this test, realistic think time settings and cache settings are not required as the objective of this test to know the server breakpoint and how it fails.

Endurance (Stability) Tests needs to be run atleast for 10-12 hours in order to identify the memory bottlenecks. This need not be run for peak load, but it can be run for average load levels.

Does Maths play a vital role in Performance Testing

2007-11-22T12:55:00.002+05:30

Most of you must be puzzled to know Maths (Statistics) plays a vital role in Performance Testing. Yes, we must accept the fact that performance tester should use mathematical knowledge to conduct effective performance testing activity.

Slowly , (i mean it took a long time for us to understand and appreciate that mathematical knowledge is required for Performance testing) we started appreciating the fact that - a new strategical move is there in Performance testing which is nothing but the use of more & more analytical approaches to model the system more realistically and to predict the system performance for varying higher loads.

I would like to share this paper - A strategical move towards Analytical modelling with you. This paper will help you to appreciate the importance and relevance of analytical approaches for effective performance engineering activities. I prepared this paper for STEPIN 2008.

Look for the entire article here...

Capacity Planning - An ITIL Way

2007-11-10T12:47:00.002+05:30

I prepared a paper for QAI - ITSM conference. Though it was not selected for the conference, just thought of sharing that paper with you.

This paper will provide you the glimpse of what is ITIL , what is ITSM , What is Capacity Planning, who to assure 100% system availability through effective capacity planning and how to do effective Capacity Planning.

Look for the entire article here...

Hope it helps!!!! Happy learning

A Paradigm Shift in Performance Testing

2007-09-18T15:32:00.000+05:30

Scott Barber shares his experience in WOPR6 (sixth meeting of the Workshop on Performance and Reliability), where he happened to meet couple of great people who are known for their work in the Performance Testing field in his August 2007 column. (http://www.perftestplus.com/resources/023PeakPerf.pdf)

He happened to meet lot of great experts including Neill McCarthy, a top performance tester from U.K,Antony Marcano, a performance test consultant in U.K and manager of Testing Reflections.com,Goranka Bjedov senior Software Engineer at Google,Craig Rapin, a performance test manager of world's largest financial services,etc where he got lot of insights about Performance testing and how its gaining maturity day by day.

Am sure that if you read his article on 'Performance Testing Moves toward Maturity' , you will really feel somewhat great to think that you are also in the great research area into which there are already 10-15 pioneers who are doing great things.

Am sure that you will feel motivated.... Now the question is how are you going to withstand with this maturity.. Yes its high time for us to act immediately and to show the maturity in our way of working too :)

Directions for fellow Performance Testers

2007-09-13T11:57:00.000+05:30

Before certifying the application performance,
are you satisfied with your own performance? are you sure that you are conducting effective performance test? Have you ever thought how to ensure the your performance test strategy and are you sure that your performance test report is bug free?

The following tips would help you to get more maturity and to do your job better

Yes..You need to be comfortable in the terminologies used in the performance engineering field and need to learn one or two popular and effective performance test tools and develop bottleneck analysis skills. That's the basic skills required for being a performance tester. Then you are matured enough to look back at your performance and look for how effective you are in your activities.

That's where, you need to start looking at Analytical modeling and simulation modeling approaches to move to your next level of expertise.

Foremost thing is that, understand the operational laws and start applying those laws in your day to day test activities.

Secondly start learning about Workload Modeling concepts and Statistical distribution fitting and apply those learning's to define performance testing goals/SLAs.

Thirdly start learning about Demand planning and Performance prediction techniques. Learn Mean value analysis and Erlang's method.

Fourth step is start learning about Capacity Planning.

This is the second level of maturity you need to gain. The next advanced third level is moving towards simulation modeling and advanced capacity planning.

Hope this helps to set your carrer path and plan to work towards it.

What is Capacity Planning in a bird's eye view?

2007-09-12T17:57:00.001+05:30

Though the awareness about the Performance Engineering is slowly increasing, still people are thinking that Capacity planning as an alternative to performance testing or sometimes planning for the server capacities before baselining the software performance.

Capacity Planning helps in identifying the specific point of time in future through prediction when the system saturation is going to occur and provides the most cost-effective way of delaying system saturation as long as possible.

Capacity Planners help in identifying the system's saturation point. When and why the saturation would occur and which are the best cost-effective alternatives for avoiding the saturation.

Before looking at the underlying server hardware, we need to first look at the software performance and perform tuning activities to get better performance. Then we need to think about the server capacities.End of the day, its the server's workload pattern which consumes the system resources. Identify the system workload and analyze the system resource usage for the typical workload expected on the server. Based on this, arrive at the capacity baseline on the current server hardware.

Based on the business forecasts, do a prediction on how much increase would be expected in the workload due to the business forecast and identify the required server resources to support the demand. Thus Capacity Forecast is done for the forecasted load expected by the business.

Things to note before you attend interview for Performance Testing role

2007-09-12T17:57:00.000+05:30

Based on my experience as a Interviewer, i would like to share few things which the interviewee must follow/know before attending the interviews for Performance tester role.Couple of things which i felt its really missing in many interview candidates is what i have compiled and provided below. I feel that the below skills are mandatory for any performance tester for 1-2 years of experience.

1. Understand that LoadRunner/Silk Performer knowledge is not everything. Its just a tool which enables us to do better performance testing.

2. Be good in Performance testing terminologies - Types of tests and its objectives,various Performance metrics,monitors,etc. The differences/changes required in scheduling each type of tests should be known.Irrespective of whether you have the responsibility for deciding the test strategy , always know when and why to conduct each of the tests.

3. First, be clear on the objective of conducting performance test for any application. I mean set your own objective that you (along with the team) need to make the system to meet its performance objective. Plan realistic tests accordingly as per the business understanding and try your best to identify various load situations that make the system to fail and work as a team to resolve those bottlenecks. Its not enough to conduct some load tests and say that application doesn't meets it Throughput SLA or response time SLA.How you identify and isolate the bottleneck is more important. What and How you observe and interpret during the test is very important.

4. As a Performance tester, you should know the basic relationship between Hits , Response time , Throughput, etc. This forms the basic for the bottleneck analysis. Increase in response time is not a bottleneck you have identified, its the symptom for some bottleneck. Its a must to identify the hidden bottleneck rather than thinking that's its out of scope of your testing. Its your responsibility to run various isolation tests to gather observations about system performance and provide your suggestions to the DB admins and system admins. Dont forget that its your responsibility to get those people involved to help you out in isolating the issues. I agree that, its a team effort but the sole leader here is the Performance tester who need to coordinate all the other members to get the bottleneck identified. Always think that they are going to help you rather than thinking that you are helping them. That thought makes lot of difference in the ownership you have and the responsibilities you hold.

5. If you are an expert in a specific performance test tool, rank your competency appropriately. Explain what you know and what you don't know. State the reasons for what you don't know and why couldn't use\know that. All the organizations will not look for tool expert who knows everything about a specific tool. Having the tool knowledge is important, but that's not everything. So, be frank enough to tell what you know and what you don't know. How much you are conceptually good is what matters in most of the cases.

6. Atleast in Performance Testing/Engineering field, i definitely feel that a good performance engineer should have referred lot of books/articles/white papers if he/she is really looking for a way to conduct more effective performance tests. Talk about how it made a difference in you/your project.

7. On the whole, the performance tester should include the tool knowledge(1 or 2 tools),knowledge and experience for doing performance requirements analysis to set performance SLAs, Test environment creation details , Bottleneck analysis skills, Basic knowledge about the technology used in the applications and basic knowledge about the OS , Server software & hardware details. (I have just stated the core required skills).

Hope it helps you to relook at your skills and identify the gaps before you attend your next interview.

How to conduct effective Performance Tests

2007-08-27T10:32:00.000+05:30

Have you thought about statistical distribution fitting of your application's user request arrival pattern ?

Check out the Posters(3 slides) and the paper titled "The Forgotten facts in the workload modeling of an ebusiness application" published in the QAI conference - STC 2007 held at Bangalore on Aug 24-25 2007.

This paper pin points few of the most forgotton facts in the workload modeling of an web application.

Look for the entire article here...

Poster Slides Presented in th QAI Conference - Poster Session
Slide 1 :

Slide 2:

Slide 3 :

Performance Testing -Begineers Reference

2007-08-19T15:32:00.000+05:30

For any one who is very new to Performance testing or who is just into performance testing and want to know more......

Scott Barber's Articles/Series about User Experience and Best Practices in performance Testing would help you to set your directions. (http://www.perftestplus.com/pubs.htm)

Infact i started my own way from Scott Barber's above mentioned series. I have read them atleast 3-4 times completely and even now whenever i find time, i still love to read them again as i feel i could get some ideas each time i read them again and again.

Give it a try and track your improvement. If it doesnt works , then write to me. I can help you to make you realize what went wrong.

A Book on Performance Testing for beginners....

2007-08-10T10:10:00.000+05:30

Thanks a lot viewers. Thanks for your encouragement by your kind mails.

I am planning to publish a book on Performance Testing comprising all that i learned during the past 3 years of the experience in my career.

I started my carrer as a functional tester and pushed or pulled into Performance testing fortunately thereby i came to know about this area. Mine is a hard way and i didnt had much support in terms of helping hands to get to know about the fundamendals before i started handling the projects. Slowly i learned a lot and finally ended up in unlearning a lot because of the ignorance i had initial days.

The objective of this book is to help fellow performance testers so that they can spend their quality time in achieving great things rather than reinventing the wheel :)

"Performance Testing Handbook - A Comprehensive Guide for Beginners"

The bird's eye view of topics covered in this book includes,

1. Why Performance Testing is crucial.
2. Dynamics and myths in performance testing
3. Deriving Quantitative performance test goals
4. Common mistakes in performance testing which i learned
5. Overview of performance test tools - How to choose the best tool
6. Setting up the performance test environment
7. Comprehensive support for writting test scripts
8. Application benchamarking
9. Quick look at the Industry standard benchmarks
10. A helping in setting up the performance Test Strategy
11. Performance Test Monitoring
12. Performance Test Execution - When and how to choose various tests (Load, Stress, Volume, Soak, Endurance tests).
13.Bottlenecks identification strategy
14.Reporting - How does it matters so much
15.Road Ahead - towards mathematical modeling and queuing theory basics

Web Log Analysis

2007-08-09T15:32:00.000+05:30

A typical issue which i faced during web log analysis...

While conducting Performance testing for applications which are falling into the category of Post-Production case, the most important activity that needs to be done is web log analysis. The web server logs are the gold mines which hold all the information about the server during the load condition.

During the web log analysis, look for the peak hour traffic time and identify what is the server load in terms of hits/sec and in terms of users. Based on this statistics, derive performance test goals by considering the business forecasts.

I would like to share a typical issue which I faced while doing web log analysis for a typical post production category of an application. The application under test uses a shared web-app server (IIS) in which there was 2 other applications deployed. Now, the problem is the web server log files have the traces of all 3 applications together as these files are common for a server and not specific for each of the applications running on the server.

The web log analyzer tool which I was using doesn’t have any option to set filters. In that case, I tried using a web log analyzer tool (123LogAnalyzer tool) which provides a filter option to filter out the traces pertaining to a specific application and export the filtered log file into a text file. For example, if the application under test has a context root as ‘methodologies’ (http://132.23.34.21/methodologies/......), then try setting up this URL in the filter and create the text files for your application of interest.

Now you could use the newly generated text file to feed it into any of the log analyzer tool which you are confident enough and proceed with your performance goal setting. My kind advice is that before choosing the log analysis tool of your choice, analyze the accuracy of log analysis thoroughly.

For more details on the various web log analysis tools, look for the web log analysis comparison article available in my blog.

Hope it helps!!!

Proactive Vs Reactive Approach in Performance Testing

2007-07-04T10:48:00.001+05:30

Reactive Approach

The Performance Testing activity is often considered as a reactive way of performance management. In most of the cases, the system performance is never thought of during the early phases of Software Development Life Cycle phases (SDLC). Performance is often thought of only as a last activity after the System Testing phase.

And most of the time , the performance bottlenecks revolves around the system architecture or the system design which is a very high cost activity to think about & in certain cases , the system is put into trash because of the huge deviations in the performance constraints.

Basically waiting for the performance problems to appear & then dealing with it at the end is always not a better approach. Mostly Performance Testing is considered as a reactive approach as there is not much importance is given to the system during early life cycle phases. It is more a ‘fix-it-later’ approach which is not that effective.

Proactive Approach

The Proactive approach anticipates the performance problems well in advance & adopts techniques for mitigating them. The Importance of the Performance is thought about from the Initial phase of Software Development Life Cycle (SDLC) & various performance engineering activities are identified for the system.

The disadvantages of ‘fix-it-later’ approach are well understood & engineering practices are adopted to analyze the system design in performance angle.

The integration of performance engineering activities with the SDLC phases is provided in the below table.

Tips for developing effective Performance Test Scripts

2007-07-04T10:48:00.000+05:30

The Test scripts are generated to mimic the user behavior during the performance tests. The test scripts enable the performance tester to simulate the real time user behavior by providing the configuration options. The Test scripts needs to be modeled in such a way that it exactly simulates the user actions & user think times.

The Test scripts should have the transaction names or browser requests identified for which the response time & throughput metrics needs to be collected. These requests should be named with proper naming convention so that it becomes easy to refer to a specific transaction by its unique name.

Running the script containing the right set of transactions alone doesn’t mimic the real time user behavior. Each transaction should be always coupled with another important metric called as think time, which is the time taken by the user to think and to navigate between various pages. This wait time which is placed between each transaction determines the accuracy of the scripts simulating the user behavior. Analyzing the think time between each transaction is very critical as it determines the load on the server at any point of time. By conducting discussions or surveys with business analysts & end users of the application, the think time for each transaction can be identified. Say, you have 2 transactions in your script, Login transaction followed by the think time of about 10 seconds followed by the Logout transaction. If the think time is changed to 5 seconds, then the frequency of the login & logout requests send to the server increases, hence the load (requests/sec) on the server increases. If the think time is changed to 20 seconds, then the server load (requests/sec) decreases as the requests are sent in 20 seconds interval.

The test data parameterization is another important activity which needs to be taken care in the scripts. Care should be taken so that enough test data is provided in the script so that each test data is not used more than once when the tests are run. For example, the user login credentials needs to be parameterized in the script. If the test is planned to run for 100 users, then the script should have 100 login credentials available in the script. If more than 1 user is allowed to use the same test data, then the response time would be varying as it might be caching the web pages or the session might get reused. It’s always recommended to make sure that the script is supplied with enough test data to be used uniquely for each user. By using parameterization, we can make sure that the correct formatted unique data is sent from the client-side for each user during the course of the test run. From the server side, there might be some dynamic data generated for each user which may be unique for each user. For example the session id of each user would be unique. If the server passes the session id of the user while returning the response for a specific client side request, then this part of the script cannot replay correctly if it is run for a different user id from the parameterized test data.

This kind of server-side dynamic data needs to be handled in the script as well. Sometimes the recorded script cannot replay because of some specific dynamic values which are generated by the server. This kind of dynamic values needs to be first studied whether it’s unique with respect to each logged in user or unique for each run even for the same user. Sometimes the user action gets completed even if the dynamic values are not handled. Hence, first analyze the logic under which there is a dynamic value getting generated from the server end & accordingly handle the dynamic value. Most of the time the dynamic value is handled by identifying the corresponding left & right boundary & capturing the dynamic value, then the captured value can be substituted against the variable which is holding the dynamic value.

Last but not least, make sure the script is very realistic in terms of the user actions. Try to have a single script which covers all the flows of the application & use random number generator function to distribute the user load across various functional flows. Sometimes it’s good to have multiple scripts as long as the distribution of the load across scripts is handled separately.

Happy Scripting!!

How to run effective performance tests

2007-06-29T10:48:00.000+05:30

To plan for an effective performance test, follow the below four important steps

1.The first step is to understand the web site’s actual usage pattern. Mostly this information can be collected from the web server log files which are considered as the ‘gold mines’ to know every possible detail about the server load. The ‘log analyzers’, tools which helps to analyze the web log files and provide the analysis report in terms of number of user sessions, number of unique users, number of errors, peak day traffic, peak hour traffic, user request arrival pattern, frequently accessed pages, user navigation pattern, etc. From this analysis, the actual usage of the server can be identified.

2.The second step is to design the performance test approach. Based on the business forecast of the user load increase, develop service level agreement (SLAs) that has to be met by the application in order to handle the user load in the target environment. The analysis done in the first step helps to set quantitative SLA for the application. Design the test cases that needs to be tested along with the parameters like ramp up strategy, test duration, think time settings, test scenarios to be tested, different load conditions & types of tests along with the pass/fail criteria. Baseline the test approach & use this as the reference for the rest of the activities.

3.The third step is to identify the performance test tool to help us in simulating the user load on the server. Look for performance test tools available in the market (Licensed or freeware tools) based on the budget constraints & technologies supported by the test tool. Using the test tool of choice, develop the test script which simulates the actual user access pattern. Have enough test data available in the test script in order to simulate different users using different data available in the script.

4.The fourth step is to look for the readiness of the test environment. The test environment includes the application’s server infrastructure & also the load test environment. Enough care should be taken to ensure that no other users access the application during the test run & the servers are isolated from any other usage. In the case of load test environment, ensure that load generators are in place to generate the required load, the availability of test tool license to support the load to be simulated, test scripts are developed and ready with the required test data & think time, server monitors are configured with the required performance counters & the server admins are informed for their assistance in case of server failures.

Choose right set of test scenarios to schedule the performance test

2007-05-26T15:31:00.000+05:30

Like functional testing, performance testing cannot be done for all scenarios. Identify the frequently used scenarios based on discussions with the business analyst. Sometimes, there will be couple of scenarios which have some suspections about its performance. We have to include those kind of scenarios too. It boils down to Pareto analysis – the system performance will be improved if 20% of the scenarios which are most frequently accessed are tuned. Tuning the remaining 80% of scenarios will not bring considerable performance improvement. Use Scott barber’s UCML or CBMG modeling to identify the critical scenarios.

During the performance test, don’t forget that you need to create the actual real time system usage model. (i.e.) The test should be very realistic in such a way that it models the real time traffic & more important is it should model the transaction mix of the real time workload. If you are scheduling a test for 100 users, then make sure that the 100 users are distributed across various functional flows of the application (Say in a banking application during the peak hours the users are distributed across various functionalities like, 60% of the users depositing the amount, 20% of the users transferring the amount, 10% of the users checking for the account balance & 10% of the users accessing the mini-statement). Unless this kind of mixed mode scenario is adopted for the testing, we might not end up simulating the real time user load.

There are certain situations wherein we might need to schedule the performance test with a single functional flow. One situation is when the performance testing is scheduled for each of the functionally stable build. Other situation would be when we identify some performance issues during the mixed mode scenario, then in order to isolate whether the performance issues are from a specific functional flow, we might need to run single functional flow test.

Observe the differences quoted above. As long as your objective is to test the system performance for a specific load, make sure the load is very realistic to the real time user access.

Metrics to be collected during any Performance Tests

2007-05-25T09:58:00.000+05:30

Irrespective of the kind of performance test (load test/stress test/volume test/spike test etc), there are certain metrics which needs to be collected from the performance test tool to analyze the system performance.

End of the day , we want to know whether the system is able to handle the load meeting the response time level SLA. To achieve that we should identify the following 2 things.

1. Whether my software pipe is able to effectively utilize the available hardware pipe.

2. Whether my hardware pipe is big enough to be used by the software pipe.

If the software pipe is not able to utilize the hardware pipe effectively, then no wonder that even if you upgrade the server hardware, there will no improvement in the system performance.

Metrics to be collected

1.As a first step always look for the average & 90th percentile response time of all the transactions. Just neglect if there is other transactions which are inserted by the tool itself. (In case of Load Runner, the tool provides the transaction response time for vuser_init, vuser_end & action transactions depending upon the run time settings).
2.Next look for the load generated from the test tool. Hits/sec is the metric which provides the information on how many requests are fired to the server per second. Look for the average hits/sec & if the test is run using a specific ramp up rate, then measure the hits/sec value during the stress period of the test(when all users where running).
3.Next look for the Throughput, the metric which portrays the load handled by the server. The total transactions passed per second metric helps to identify the how capable is the server to handle the transactions. Also look for throughput in terms of bytes/sec metric which again portrays the server capability to respond to the client requests.
4.Next look for the failures occurred during the test. Observe when the failures have occurred, what is the type of failures (network failure, HTTP 500 errors, HTTP 404 errors, proxy server failures, etc) & is the failures specific for a transaction.
5.Next, look for the basic performance counters of web/application/DB server resources like %CPU, processor queue length, Memory pages/sec, Page faults/sec, % Disk utilization.

These five are the basic performance metrics which are mandatory for any performance tester to analyze the performance of the system. Going forward, in order to isolate the performance bottlenecks, we might end up looking at various performance metrics depending upon type of the performance issue.

Know the importance of Think time

2007-05-20T14:46:00.000+05:30

In most of the performance testing approach, the ‘Think Time’ metric is taken for granted. We should understand the importance of think time before deciding the test strategy.

What is a Think Time? Think time is the time taken by the users to think or to navigate
to different pages in the application.

Depending upon the application context, the think time would vary. Its not advisable to have the default think time for all applications under testing. For the user registration page containing 5 mandatory fields, the use might take 15-20 seconds on an average. Notice the words “15-20 seconds on an average”. You can’t say that all the users would take exactly a specific think time. It would definitely vary based on each of the users. So, always make a point that whenever you run the load test, apply a randomization factor on the configured think time. Say if the think time in the script is 10 seconds between each of the transactions, configure the load test tool to use a randomized think time of 80% to 120% of the value available in the script.

As we are interested to measure the response time of the transaction (user defined actions), care should be taken to place the think time outside the transaction points. (i.e.) the think time should not be placed between the start & end of the transaction.

Most of the testers doing the performance testing don’t understand that by changing the think time, we can simulate the load of large number of users though we run the test for less number of users. To illustrate, consider the following example. Five tests were conducted with 5 virtual users simultaneously accessing the application. In each of the test, the think time is the only metric which was changed. Observer, the Hits/sec generated from the client side & Throughput (the number of transactions passed /sec).

Though the number of users hitting the server is same, the think time creates a huge impact on the server load. We need to understand the fact that decreasing the think time will increase the server load & vice versa. Always try to model the real time scenario & identify the performance figures. Never forget that we can play around with the server load just by changing the think time.

Also, dont forget to use Little's law, N = X . (R+Z) to check whats the actual load on the system for the test which you are running.

Relationship between Metrics

2007-03-21T10:57:00.000+05:30

Things to note : Queuing Theory basics

1. The utilization of a service center is dependent on the Arrival rate.
2. The Service Demand remains constant irrespective of the load (for Load Independent devices).
3. Throughput increases with the Utilization.
4. Queues will be formed even when the utilization is less than 100% & its also possible that there can be no queues during 100% utilization too. Hence there is no relationship between the Utilization & Queue length.
5.The response time does not depend on the Utilization. It depends on the Queue length, hence the arrival pattern.
6. The maximum CPU utilization that can be achieved can be calculated using the formula Queue length / Queue Length + 1.
(i.e) For a four CPU m/c maximum utilization that can be achieved = 4/5 = 80%

Importance of Little's Law

2007-03-06T12:23:00.000+05:30

Little's law is quite simple and intuitively appealing.

The law states that the average number of customers in a system (over some time interval), N, is equal to their average arrival rate, X, multiplied by their average time in the system, R.

N = X . R (or) for easy remembrance use L = A . W

This law is very important to check whether the load testing tool is not a bottleneck.
For Example, in a shop , if there are always 2 customers available at the counter queue , wherein the customers are entering the shop at the rate of 4 per second , then time taken for the customers to leave from the shop can be calculated as

N = X. R

R = 2/4 = 0.5 seconds

A Simple example of how to use this law to know how many virtual users licenses are required:

Web system that has peak user load per hour = 2000 users
Expected Response time per transaction = 4 seconds
The peak page hits/sec = 80 hits/sec

For carrying out Performance tests for the above web system, we need to calculate how many number of Virtual user licenses we need to purchase.

N = X . R
N = 80 . 4 = 320

Therefore 320 virtual user licenses are enough to carry out the Load Test.

Kendall's Notation

2007-03-05T10:39:00.000+05:30

In Queueing Theory , Kendall's Notation is used to describe or classify the queuing models.

A / B / C / ( D / E / F )

A - It represents the arrival pattern of the requests to the system.

B - It represents the servicing pattern the system.

C - It represents the total number of servers in the system.

D -It represents the total number of requests that can be queued in the system.

E -It represents the calling source (i.e) the customer base

F -It represents the service discipline of how the job that gets served by the system.

Example :

1. M/M/1/20/30/FIFO = markovian arrival rate , markovian service time , 1 server queuing model , maximum queue length as 20 , user base as 30 , service discipline as First In First Out (FIFO).

Fundamental laws of Performance Testing

2007-02-28T17:18:00.000+05:30

Sometime back , i could realize how I was struggling to isolate simple Performance bottlenecks.

Though i managed to identify performance bottlenecks,I know that i was missing something..but really not sure what was it. Pea (Performance Engineering Associates) helped me to identify those missing pieces. Thanks to Pea.

Yes..Its the basics about Queuing Theory.

PEA was formed by professionals who have had the opportunity to lead assignments that brought about several radical changes by bridging the gap between theory and practice in PE.

Fundamental Laws
~~~~~~~~~~~~~~

1. Utilization law
2. Little's law
3. Response Time law
4. Forced Flow law
5. Service Demand law

Check out the free online book on Queueing Theory available in pea's web site.

http://www.pea-online.com/resources.htm

Nice article from the book "Performance by Design - Computer Capacity Planning by Example" by Virgilio A.F. Almeida, Lawernce W.Dowdy, Daniel A.Menasce.

http://www.informit.com/articles/article.asp?p=170721&seqNum=2&rl=1

Watch out for more information about Performance Engineering Vedas shortly.

Learn about Performance Testing from Scott barber's Site

2007-02-28T17:04:00.000+05:30

Who said Performance testing is just simply simulating more number of users to test the system performance.......

There are lots more than that......Words cant explain the worth of information available in Scott Barber's site.

http://www.perftestplus.com/pubs.htm

Starting from how to plan for a Performance Test till how to create the Performance Test Report are available in a very easy & understandable way. Its amazing.

Thanks to Scott barber for sharing the information with us.

Performance Test Estimation Approach

2007-02-27T11:45:00.001+05:30

Challenge your Performance Test Estimation Effort Using Test Factors

Performance Test effort estimation is the initial challenge we face in the Performance Test Life cycle. As Performance Testing is more like an investigation activity which may take any direction, the ad-hoc estimation methodologies used for performance test estimation doesn’t seem fruitful. Because of the dynamism and varying scope, there are lots of issues in the test effort estimation. Based on our experience in the performance testing projects on the web based applications, we conjecture that the Performance Test Estimation approach using Test Factors could be more reliable. Our approach is based on the consideration of the various test factors involved in scripting, execution and analysis, thereby eliminating the use of ad-hoc estimation methodologies and it is architecture / technology independent.

This paper describes how to do effort estimation for a performance testing project. The paper focuses on the pre-requisites and the factors which need to be analyzed in order to arrive at the performance estimation model along with the details about how to use this estimation model. This approach is being implemented in Honeywell for the estimation of Performance Testing effort on web applications and is proven to be ninety-five percent accurate. It’s been used to estimate ten projects falling under simple, medium and high complex project categories. The case study illustrates the application of this technique to arrive at the estimation for a typical web application.

The readability of the paper covers Performance Test Engineers, Test Leads, Test Managers and others with the domain knowledge of performance testing.

Look for the entire article here...

Network Bottleneck Symptoms

2007-02-26T14:42:00.000+05:30

The simplest way to measure effective bandwidth is to determine the rate at which your server sends and receives data. Network bandwidth availability is a function of the organization's network infrastructure. Network capacity is a function of the network cards and interfaces configured on the servers.

Network Interface: Bytes Total/sec : To determine if your network connection is creating a bottleneck, compare the Network Interface: Bytes Total/sec counter to the total bandwidth of your network adapter card. To allow headroom for spikes in traffic, you should usually be using no more than 50 percent of capacity. If this number is very close to the capacity of the connection, and processor and memory use are moderate, then the connection may well be a problem.

Web Service: Maximum Connections and Web Service: Total Connection Attempts : If you are running other services on the computer that also use the network connection, you should monitor the Web Service: Maximum Connections and Web Service: Total Connection Attempts counters to see if your Web server can use as much of the connection as it needs. Remember to compare these numbers to memory and processor usage figures so that you can be sure that the connection is the problem, not one of the other components.

To determine the throughput and current activity on a server's network cards, you can check the following counters:

· Network\Bytes Received/sec
· Network\Bytes Sent/sec
· Network\Bytes Total/sec
· Network Current Bandwidth

If the total bytes per second value is more than 50 percent of the total capacity under average load conditions, your server might have problems under peak load conditions. You might want to ensure that operations that take a lot of network bandwidth, such as network backups, are performed on a separate interface card. Keep in mind that you should compare these values in conjunction with Physical Disk\% Disk Time and Processor\% Processor Time. If the disk time and processor time values are low but the network values are very high, there might be a capacity problem. Solve the problem by optimizing the network card settings or by adding an additional network card. Remember, planning is everything—it isn't always as simply as inserting a card and plugging it into the network.

Disk Bottleneck Symptoms

2007-02-23T10:45:00.000+05:30

A bottleneck from a disk can significantly impact response time for applications running on your system.
Physical Disk (instance)\Disk Transfers/sec counter for each physical disk and if it goes above 25 disk I/Os per second then you've got poor response time for your disk.
By tracking Physical Disk(instance)\% Idle Time, which measures the percent time that your hard disk is idle during the measurement interval, and if you see this counter fall below 20% then you've likely got read/write requests queuing up for your disk which is unable to service these requests in a timely fashion. In this case it's time to upgrade your hardware to use faster disks or scale out your application to better handle the load.

Look for the Physical Disk (instance)\Average Disk Queue length & Physical Disk (instance)\Current Disk Queue length parameters to get more details on the queued up requests.

CPU Bottleneck Symptoms

2007-02-23T10:33:00.000+05:30

Symptoms for CPU bottlenecks include the following :

The Processor(_Total)\% Processor Time(measures the total utilization of your processor by all running processes) will be high. If the server typically runs at around 70% or 80% processor utilization then this is normally a good sign and means your machine is handling its load effectively and not under utilized. Average processor utilization of around 20% or 30% on the other hand suggests that your machine is under utilized and may be a good candidate for server consolidation using Virtual Server or VMWare.

Further to breakdown this %processor Time , monitor the counters - Processor(_Total)\% Privileged Time and Processor(_Total)\% User Time, which respectively show processor utilization for kernel- and user-mode processes on your machine. If kernel mode utilization is high, your machine is likely underpowered as it's too busy handling basic OS housekeeping functions to be able to effectively run other applications. And if user mode utilization is high, it may be you have your server running too many specific roles and you should either beef hardware up by adding another processor or migrate an application or role to another box.

The System\Processor Queue Length(indication of how many threads are waiting for execution) consistently greater than 2 or more for a single processor CPU is a clear indication of processor bottleneck . Also look at other counters like ASP\Requests Queued or ASP.NET\Requests Queued as well.

Code & Application server related performance issues in J2EE

2007-02-23T10:30:00.000+05:30

In J2EE environment, there are some common Code related or Application server related problems. It include :

Code related problems :
~~~~~~~~~~~~~~~~~~
1. Slow Methods
a. Consistently Slow Methods
b. Intermittently Slow Methods
2. Synchronization Problems
3. Memory Problems
4. Coding Practices, such as using exceptions as a means to transfer control in the applications

Application server configuration problems :
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1. JDBC Connection Pool size
2. JVM Heap size
3. Thread Pool size

Symptoms for Application & Web Server Bottlenecks

2007-02-22T17:28:00.000+05:30

Symptoms for Application server bottleneck
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1. Increased 'Server Time' breakup
2. One or more page components of transaction takes more time where in the DB query is having less execution time.
3. The Static files are having less response time whereas the dynamic contents (servlets,jsp,etc) takes more time.
4. Network delay is negligible.
5. Home Page gets displayed in few seconds even during the stress period(as it is fetched from the web server).
6. Hits / sec & Throughput remains less.
7. If the CPU/ Memory/Disk of the App server has any bottleneck symptoms.
8. If the HTTP / HTTPS connections established doesn’t increase proportionally with the load.
9. If the new connections established is very higher & the reused connections are very less.

Symptoms for Web server bottleneck
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1. Increased 'Server Time' breakup
2. One or more page components of transaction takes more time where in the DB query is having less execution time.
3. The static files are having high response time than the dynamic contents (servlets,jsp,etc).
4. Network delay is negligible.
5. Home Page takes more time for display.
6. Hits /sec in the web server is very less.
7. If the CPU/ Memory/Disk of the web server has any bottleneck symptoms.

A Comprehensive Report on Web Log Analysis

2007-02-22T14:05:00.002+05:30

In today’s dynamic world, the IT sector of all the businesses want to use more sophisticated techniques to support their application scalability. All the web site owners are interested to know the usage pattern of their site in order to make any business decisions. The necessity to know the visitor profile is very high in any web site as most of time the visitors are anonymous & unpredictable for any open systems (which uses internet).

The Web analytics activity deals with measuring the end user behaviors on a web site. Of the two major web analytics techniques (Web server log analysis & Page Tagging) available, this paper deals with the Web Server Log Analysis concepts. This paper provides an overview of various log analyzer tools available in the market & their comparative ratings.

The Web log analysis is used by various types of users with different backgrounds. The different sets of users are interested to know about specific data from the web server log. This has lead to the development of lot of log analyzer tools with specific extra features, though the basic features are available in most of the tools.

Look for the entire article here...

Hardware Malfunctioning Symptoms

2007-02-22T10:47:00.000+05:30

System\Context Switches/sec (measures how frequently the processor has to switch from user- to kernel-mode to handle a request from a thread running in user mode.). If this counter suddenly starts increasing however, it may be an indicating of a malfunctioning device, especially if you are seeing a similar jump in the Processor(_Total)\Interrupts/sec counter on your machine.
You may also want to check Processor(_Total)\% Privileged Time Counter and see if this counter shows a similar unexplained increase, as this may indicate problems with a device driver that is causing an additional hit on kernel mode processor utilization.
If Processor(_Total)\Interrupts/sec does not correlate well with System\Context Switches/sec however, your sudden jump in context switches may instead mean that your application is hitting its scalability limit on your particular machine and you may need to scale out your application (for example by clustering) or possibly redesign how it handles user mode requests. In any case, it's a good idea to monitor System\Context Switches/sec over a period of time to establish a baseline for this counter, and once you've done this then create a perfmon alert that will trigger when this counter deviates significantly from its observed mean value.

Memory Bottleneck Symptoms

2007-02-22T10:41:00.000+05:30

When it comes to the System memory , there are 3 things to monitor :

1.Monitor Cache (Hits/Misses),
2.Monitor Memory (Memory Available/sec, Process/Working Set),
3.Monitor Paging (Pages Read/Sec, Pages Input/Sec, Page Faults/Sec, % Disk Processing)

Memory\Available Bytes, if this counter is greater than 10% of the actual RAM in your machine then you probably have more than enough RAM and don't need to worry.

The Memory\Pages/sec counter indicates the number of paging operations to disk during the measuring interval, and this is the primary counter to watch for indication of possible insufficient RAM to meet your server's needs.

You can monitor Process(instance)\Working Set for each process instance to determine which process is consuming larger and larger amounts of RAM. Process(instance)\Working Set measures the size of the working set for each process, which indicates the number of allocated pages the process can address without generating a page fault. A related counter is Memory\Cache Bytes, which measures the working set for the system i.e. the number of allocated pages kernel threads can address without generating a page fault.
Finally, another corroborating indicator of insufficient RAM is Memory\Transition Faults/sec, which measures how often recently trimmed page on the standby list are re-referenced. If this counter slowly starts to rise over time then it could also indicating you're reaching a point where you no longer have enough RAM for your server to function well.

A Paper on Risk based Testing

2007-02-21T13:59:00.000+05:30

Dare to Gamble - Creating a new vibe in Risk Based Testing

Abstract

Uncertainty introduced by the testing method is virtually unavoidable. Testing activities can fail in many ways; however, you can prevent most problems through Risk analysis in the early test life cycle. Risk implies a potential negative impact on any of the test activities affecting the system either directly or indirectly. Steve Wakeland defines IT risk as ‘the likelihood that a program fault will result in an impact on the businesses’.

Risk based testing differs from the traditional way of system testing because it emphases on the various risks involved in the testing life cycle. Though the probability of the risk is known, the outcome remains unknown and hence we could define Risk as the product of the defects and probability.

The objective of our paper is to put forth the risk based test strategy which would help testers to identify the risks involved at the earlier stages of a test life cycle and the reduction of COPQ. The paper deals with the risk prediction and mitigation methodologies. The Metrics Based Management approach illustrated in our case study enabled us to identify the probability and the consequences of individual risks during the test life cycle.

A risk analysis was performed and the functions with the highest risk exposure, in terms of probability and cost, were identified. Risk prediction is derived from planning, assessing and mitigating risks. Risk prediction involves forecasting risks using the history and knowledge of previously identified risks. The risk mitigation activity is done to inspect and focus testing on the critical functions to minimise the impact a failure in the function.

The readability of the paper covers test engineers, test leads and test managers and others with the knowledge of software testing.

1. Introduction

Testing is the time consuming part of software engineering & it is the last phase in the Software Development Life Cycle .It has to be always done under severe pressure. It would be unthinkable to quit the job, to delay delivery or to test badly. The real answer is a differentiated test approach in order to do the best possible job with limited resources.

For getting more confidence that the right things are tested at the right time, risk based testing can help, as it focuses and justifies test effort in terms of the mission of testing itself. The basic principle of Risk Based testing is to test heavily those parts/components of the system that pose the highest risk to the project to ensure that the faultiest areas are identified. At the system level, one probably has to test first what is most important in the application & secondly, one has to test where one may find most defects.

This paper provides a method for Risk Based Testing. The test approach is explained in the following sections. The section 2 details on the basics of Risk based testing; the section 3 provides the high-level test process proposed for Risk based Testing; section 4 illustrated our test strategy using a case study.

2. Risk Based Testing
2.1 Risk
Risk is a possible failure, unwanted event that has negative consequences. According to
Wikipedia , “Risk is the potential future harm that may arise from some present
action.” While talking about risk, we need to think about the cost of it & the probability that it might occur.

Risk = Cost * Probability
Normally Risks can be accepted, prevented or transferred. We need to look at the degree to which we could change the outcome. Risks are divided into three general types: project, business, and technical risks.

2.2 What is Risk Based Testing
The important question that everyone has is “What makes Risk Based Testing different from traditional testing”. We all know that money is important in life, but the importance of it is different from people to people. There are lot of differences between the people who look for job satisfaction in terms of getting a challenging work to earn money & there are people who work for organizations only because of the high compensation package. Though Risks is identified in traditional way of testing, more emphasis on the Risk & building a test strategy around the identified risk makes Risk based Testing different from the Traditional Testing.

Most risk based testing is a black box testing approach .This technique checks the system against the requirement specification. Once the high prioritized risks are identified, testing strategy needs to be developed to explore on those high priority risks.

Our approach is more based on the Risk Analysis activity model shown in Figure 1. This model is taken from Karolak’s book “Software Engineering Risk Management”.

3. Our proposed method for Risk based Testing
Our approach concentrates on the project level technical risks The Business level risks & Management level risks identification/tracking/mitigation is not a part of our approach. This section provides an overview of the various steps involved in the Risk based Testing methodology. The Case study provided in the section proves as the Proof of concept for the below described model.

The Test strategy includes the following steps:

3.1 Risk Identification

3.2 Risk based Testing strategy

3.3 Risk Reporting

3.4 Risk Prediction

3.1 Risk Identification

A risk is an unwanted event that has negative consequences. It’s highly important that the risks analysis should start from the starting of the testing life cycle.

3.1.1 Requirements Risk
Normally, the software testers review the requirements documents before starting the test case designing. Most of the time the requirements stay absurd or contradictory or misleading. Lot of research is going on how to measure the quality of requirements document. Here is a simple way of reducing the requirements level risk which leads to high COPQ when uncovered during later phases of testing. The assumption of this model is that the requirements document has more than 95% coverage of the product requirements & the tester’s role is to verify the requirement document for testability.

1.The requirements should not have any TBDs or TBAs.
2.Each requirement should be rated for the understandability, clarity & testability in the scale of 1 to 5 where 5 being the idealistic value. The product of the U (understandability), C (clarity) & T (testability) factors gives the risk factor for the requirement. The minimum threshold value for the risk factor is 45. Any requirement which is having the risk factor less than the minimum threshold value needs review & revision.
3.Each Module / Component should be rated for the requirements coverage on the following areas – Functionality requirements (FR), Usability & User Interface requirements (UR), and Security & Performance requirements (S&PR). 4.The product of FR, UR & S&PR gives the risk factor for the module or component. The minimum threshold value for the risk factor is 45. Any requirement which is having the risk factor less than the minimum threshold value needs review & revision.
5.Schedule a SGR (Stage Gate Review) with the development team, testing team & the other stakeholders of the project to review & revise the individual requirements and the modules which has not met the minimum threshold value.

Most of the requirements based risks are eliminated by following the above 4 steps. All of us should accept the fact that the risks in the Requirements are the most severe.

3.1.2 Customer CTQs based Risk Matrix
The various risks (project, business, and technical risks) need to be analyzed before starting the test planning. The following are the set of activities which needs to be followed in this phase. Software Testers needs to validate & confirm the Customer’s CTQs against their system understanding.
1. The tester needs to develop a QFD (Quality Function Deployment - Six sigma tool) to prioritize the relevant Testing Types(Functional Testing, User interface Testing , Security Testing, Performance Testing, etc) required for testing the project to meet the customer CTQs(Critical To Quality).
2. The tester needs to develop another QFD to prioritize those modules/components which needs immediate testing in order to meet the customer CTQs.
With the prioritized list of Modules/Components based on customer CTQs , Customer Priority based Component Risk Matrix needs to be developed. It is similar to ‘Component Risk Matrix’ as per James Bach style, but the difference being more emphasis on the Customer expectation based Risk prioritization.

SWOT Analysis technique & Assumption risk mitigation activities help in uncovering lot of technical risks. Clear communication of the risk is as important as the Identification of risks. The Customer Prioritization based Component Risk matrix will provide the clear details on the set of risks in each component of the system.

3.2 Risk based Testing Strategy
The High risk components identified from the above activity are sorted based on the Risk Factor. This list would serve as the input for carrying out the activities in this phase.

For each Component, identify the Smoke test cases which would uncover more defects in the risk prone areas identified in ‘Customer Priority based Component Risk Matrix’. Hence the risk based smoke test case is different from the normal smoke test cases which are developed in Traditional testing.
Each Test case should carry a RPN value (Risk Priority Value). For calculating the RPN value of a test case, provide the rating against Severity of the risk , Probability of risk occurrence & Detection capability of the risk (the ease in detecting the risk). The Range is defined as 1, 5, 9 where 9 being the highest.
The Test cases with RPN values listed against the release versions can be shared with the project stakeholders.
The Management & Customers could decide on the whether specific module level test cases needs to be prioritized or the Smoke test cases with the highest RPN values.

3.3 Risk Based Reporting
Risk reporting is based on the smoke test cases (risk based) identified during the previous topic. The defects uncovered during smoke testing needs to be reported in a meaningful way. It’s important to closely monitor the errors & collect the following metrics:

· Number of Smoke Test cases planned vs. executed vs. failed.
· The test effort planned vs. actual.
· Number of Planned releases(to testing team) vs. number of actual releases
· Number of rejected builds
· Number of resources working on the project
· Total number of defects identified during smoke testing.
· Total number of defects rejected.
· Number of rejected defects due to functional changes
· Number of rejected defects due to lack of understanding
· Total number of defects per module.
· Percentage of tests passed
· Percentage of tests failed
· The effort(in hours) required to identify a defect
· The average time interval between each defect.
· Total number of defects that does not has a test case
· Total number of test cases added at run time
· Number of defects per developer(if possible)
· Number of defects per defect type
· High Priority defects found vs. Medium or Low priority defects
· Total number of deferred defects
· Metrics about Defect Injection phase & defect reason would add value

The identified defects needs to be tracked against the risk of not fixing & it would help the development team to fix the high risk defects. The defects can be plotted as shown in the below graph (Figure 2) for easy interpretation of high priority risky defects. The defects listed in the top right corner marked as “1” needs to be concentrated first.
3.4 Risk Prediction

Risk prediction is derived from the previous activities of identifying, assessing & prioritizing, mitigating and reporting risks. Risk prediction involves forecasting risks using the history and knowledge of previously identified risks.

During test execution it is important to monitor the quality of each individual function & the metrics collected during the testing phase becomes the data for forecasting future risks.

4. The Case Study
The rest of this paper will discuss a case study using the risk based approach to software testing, relating the different activities discussed in the section 3.

4.1 System under Test
The Early Indicators Product Tracking is a process to track the performance of the in-service products based on Reliability analysis. This helps in proactively identifying the in-service products that are experiencing degraded reliability performance. This results in efficient spares management and warranty issues.

The system interacts with various types of databases (SQL, Oracle, COBOL & FoxPro) & collects the various details about the aircraft parts & derives the reliability of various parts.
The system accepts the data from various vendors & does a reliability analysis & provides various charts & reports which enable the vendor in identifying reliability factor of the part.

4.2 Challenges in Testing
The following are the key challenges in the project.

1. Limited Testing Time: As the project required new complex algorithms & services to be implemented, the development timeframe was more than the expected & hence the testing phase needs to squeeze. So it’s very critical to adopt a test strategy wherein the more critical core module which represents more risk needs to be tested first.

2. Limited Testing Resource: There were not much senior testing resources available at the time of the project. The project demands more competent test engineers who have knowledge on the System architecture as the testing involves testing of simulation model & window services. It is highly essential to plan for a resource to complete the testing as soon as possible without putting in efforts for Knowledge transitions or trainings.

3. Simulation Testing: It needs a White box testing approach as most of the time, the tester needs to adopt the reverse engineering strategy to the available data & attain the data integrity.

4. Database Relativity: The system talks with 4 different types of databases (SQL, Oracle, and FoxPro & COBOL) & tracks the reliability of the aircraft parts.

5. Integration with Third party Tools: As the system exposes an interface to interact with the Third party tools like Minitab, thorough testing on the critical areas is required. The limitations of the Third party tools should not interfere in the system testing.

4.3 Risk based Test Strategy
The EIPT application consists of seven modules. The following activities are done as part of the Test Strategy.

4.3.1 Requirements Risk identification
During the requirements document review by the testing team, the requirements of the above mentioned modules which are adhoc & absurd were identified & highlighted. Thus the requirements are validated & revised as per the Requirements Risk mitigation strategy. Each of the modules which lack in functional requirements coverage or Usability & User Interface requirements coverage or Security & Performance requirements coverage were identified & highlighted by following the strategy mentioned in section 3.1. The below table (Figure 3) provides the sample of how the module level requirements are validated & risk factor is derived.

Figure 3 : Module Risk Factor Calculation
After the end of the preparing the list of requirements & module which are vulnerable to failures, Stage Gate Review (SGR) was scheduled to review & revise the requirements. By validating the requirements against the below sample points would uncover a lot of risks:

Is any tool used for creating prototypes
Configuration Management tool not being identified for placing the project artifacts
Software tools to support planning & tracking activities not being used regularly
Requirements put excessive performance constraints on the product
Unavailability of tools or test equipment
Incomplete interface descriptions
Operation in an unfamiliar or unproved hardware environment causes unforeseen problems
Strict requirements for compatibility with existing systems require more testing, and additional precautions in design and implementation than expected

4.3.2 Customer CTQs based Risk Matrix
The objective of this activity to identify the prioritized set of modules based on Customer CTQs & analyzing /identifying the risks available in those modules. Normally the risk prioritization strategies concentrate on prioritizing the risks based on risk severity, meeting the deadlines, failure mode use cases, etc. Our proposed method is found to be more advantageous than the available methods as it concentrates & prioritizes the risks based on the customer expectation & delivers quality.

The VOC (Voice of Customer) was identified by having discussion with the customer & it helped in validating the customer expectations with the QFD developed during requirements phase of the project.
Then in order to prioritize the type of testing required for meeting the customer expectation, QFD was done. The relative rating as shown in the figure 4 shows that Functional testing is the high priority as per customer’s expectation.

Figure 4 : Prioritizing Testing Type using QFD Tool
In order to prioritise the modules for meeting the customer expectations, QFD was done. The relative rating showed the high priority module that needs more concentration as shown in Figure 5.

Figure 5 : Prioritizing the Modules using QFD Tool

The Customer priority based Component Risk Matrix was developed to identify the module level risks & the Risk factor is derived in order to prioritise those risks which are of interest to the customer. For identifying the risks in each module SWOT analysis & Assumption risk analysis are used. The below table(Figure 6) provides the list of risks in each module & the corresponding risk factor.

Figure 6 : Customer priority based Component Risk Matrix

4.3.3 Risk based Test case Design

After the risk assessment & prioritization, smoke test cases were identified against each of the modules. These smoke test cases emphasis on testing the risk areas / fault prone areas of the module. Each test case carry the 3 values - Severity rating(S) , Probability rating(P) & the Detection capability rating(D) & the RPN value was derived for each of the test cases.

The below table lists the sample set of test cases of various modules with their corresponding RPN values.

Figure 7 : Risk based Testcase Design Summary

4.3.4 Risk Reporting

The smoke test cases developed during Test case design phase were executed during testing phase & there were lot of defects logged against each of the modules in a defect tracking tool.

Figure 8 : Risk Report Metrics

4.4 Our Risk based approach - Credits

Our risk based test approach has lot of advantages compared to the traditional test approach.

1. Our risk based approach consumes less resource than the traditional approach. The resource profiles of the Traditional approach, Risk based approach (planned) & Risk based approach (actual) are provided below. As risk based testing emphasis more on identifying the risk areas & uncovering the defects in those error prone areas, the time taken to test those specific core areas are less than the Traditional test effort.

Figure 9 : Resource Profile Comparison

2. The count of high priority (Critical or Catastrophic defects) defects found using the risk based approach is higher than the traditional approach. Though the total number of defects found using the traditional approach might be higher than the risk based approach, more severe defects are identified in Risk based approach as it emphasis on identifying critical defects before interim releases.

Figure 10 : Defects Comparison

3. Another success criteria of our risk based approach depends on having an efficient, effective & flexible test organization.

4.5 Limitations of our Risk based approach

The Risk based approach might not be applicable for newly formed team or inexperienced software testers as it needs more risk foreseeing/assessing competency.
Our approach might not yield great results for projects wherein the cohesion between the system modules are very high & it’s tough to isolate a risk easily.
Our approach might not be the best for projects which does not follows waterfall model of software development.

5 Conclusion

Stephen Besson says that “lowering the risk to 50 percent can be achieved in a shorter period of time if testing prioritization has been done with risk in mind ”. Risk-based testing is a highly effective testing technique that can be used to find and fix the most important problems as quickly as possible. If done well, it allows the test function to expend relatively more effort on the parts of the system that are higher business risk. These ‘higher risk’ items are also scheduled for testing as early as possible in the lifecycle. This approach is complimentary to any development lifecycle from very structured approaches (such as V-Model) through to the more dynamic rapid development methodologies (such as RAD).

6 References

1. Ståhle Amland , Risk-Based Testing and Metrics EuroSTAR99 Proceedings, 1999. http://www.amland.no/articles

2. James Bach on Risk-Based Testing Software Testing and Quality Engineering,Nov/Dec 1999.

3. Cem Kaner, James Bach, Bret Pettichord , “Lessons learned in Software Testing” Wiley, 2001.

4. Shari Lawrence Pfleeger: Risky Business: “What we have yet to learn about SW risk Management” Journal of Systems and Software, 11/2000.

5. Dale Walter Karolak, “Software Engineering Risk Management”, IEEE Computer Society Press, 1996.

6. Norman E. Fenton & Shari Lawrence Pfleeger, "Software Metrics, a rigorous & practical approach", 2nd edition, International Thomson Computer Press, 1997.

7. Linda H. Rosenberg, Ruth Stapko, Albert Gallo , “Risk-based Object Oriented Testing”

**********End of Paper**********

What is the difference between Simultaneous & Concurrent users ?

2007-02-20T12:37:00.000+05:30

These are the 2 terms which is often used in Performance Testing.

Simultaneous users are the users who have a valid session in the server. Each of the users would be performing different actions like one doing login , other viewing the reports, etc.

Concurrent users are the users who have a valid session in the server & they are performing the same operation at any point of time. All the concurrent users would perform login, reports download, etc at the same point of time.

Hence for a web site, the number of simultaneous users would be always greater than the concurrent users.For example , a banking web site has the workload of about 30000 simultaneous users & 5000 concurrent users.

Knowing the correct figures of simultaneous & concurrent users is very vital in order to perform performance testing of an application.

Learn To Win

Performance Testing ebook details

Info about New Ebook on Performance Testing

WCF Services Performance Testing

Web Services Performance Testing

Software Performance Testing Handbook - A Comprehensive Guide for Begineers

Chapter 12: Road Ahead – Moving towards advanced Performance Engineering techniques

Chapter 11: Performance Test Reporting

Chapter 10: Performance Bottleneck Analysis

Chapter 9: Performance Test Monitoring

Chapter 8: Performance Test Execution

Chapter 7: Application Benchmarking

Chapter 6: Setting up Performance Test Environment and Performance Test Scripts development best practices

Chapter 5: Overview of Performance Testing Tools

Chapter 4: Workload Identification

Chapter 3: Deriving Performance Test Goals

Chapter 2: Performance Testing – A Quick Overview

Chapter 1: Introduction

Software Performance Testing Handbook - Preview Index

My Book on Software Performance Testing for Begineers

Performance Simulation using Ptolemy Framework made easy by PEA

Do think about Think Time

Response Time Vs Queue Length Vs Server Utilization

Tips to define the Performance Test Strategy

Does Maths play a vital role in Performance Testing

Capacity Planning - An ITIL Way

A Paradigm Shift in Performance Testing

Directions for fellow Performance Testers

What is Capacity Planning in a bird's eye view?

Things to note before you attend interview for Performance Testing role

How to conduct effective Performance Tests

Performance Testing -Begineers Reference

A Book on Performance Testing for beginners....

Web Log Analysis

Proactive Vs Reactive Approach in Performance Testing

Tips for developing effective Performance Test Scripts

How to run effective performance tests

Choose right set of test scenarios to schedule the performance test

Metrics to be collected during any Performance Tests

Know the importance of Think time

Relationship between Metrics

Importance of Little's Law

Kendall's Notation

Fundamental laws of Performance Testing

Other Useful Links about Performance Testing/Engineering

Learn about Performance Testing from Scott barber's Site

Performance Test Estimation Approach

Network Bottleneck Symptoms

Disk Bottleneck Symptoms

CPU Bottleneck Symptoms

Code & Application server related performance issues in J2EE

Symptoms for Application & Web Server Bottlenecks

A Comprehensive Report on Web Log Analysis

Hardware Malfunctioning Symptoms

Memory Bottleneck Symptoms

A Paper on Risk based Testing

What is the difference between Simultaneous & Concurrent users ?