You could look at this blog as a repository of information related to performance testing n engineering areas. It provides key details from basic conceptual foundation topics & advanced topics providing a detailed insight to the readers. Am sharing some of the key things that i learnt in my career to benefit future Performance Assurance professionals to also it will provide me an opportunity to learn from your experiences. You can also reach me out through www.elitesouls.in
Hereby am publishing my first version of ebook on performance testing which was pending for a long time :). I have tried my best to compile my learnings and experience to help you out. I guess you will get benefited out of it.
Happy Learning!!!
You can download the book through the online academy @ elitesouls.in :
As discussed in earlier chapters, mere plans for Performance testing during the end of the SDLC cycle is considered as a reactive approach wherein Performance Engineering approach aims at integrating the performance engineering activities from starting of the SDLC to build a system with high performance. Thus, Performance testing forms a subset of the Performance Engineering activity. Accordingly, the competencies required for a performance engineer is little different from the performance tester.
Software Performance Testing is the act of evaluating the software system for its performance and finding the bottlenecks in the system. Software Performance Engineering is the systematic approach of constructing the software that meets the performance objectives.
Nowadays, IT companies have slowly started adopting the performance engineering activities for critical applications though 70-80% of them have adopted mere performance testing activities.
Queuing Theory is the mathematical study of waiting lines (or queues).It is interchangeably spelled as Queueing Theory or Queuing Theory. It is considered as a branch of Probability Theory and Operational Research as it is used in making business decisions. It is used in various fields like telecommunications, networks, traffic management, etc. The notation for describing the queuing model was first introduced by David .G. Kendal in 1953.
The figure represents a model of a single queue. Jobs arrive at the queue from the input source, each job bringing a demand for service from the queue. If the queue becomes full, the arriving job is not allowed to join the queue. If the job enters the queue, it may compete with other jobs for service from the queue. When a job completes its service, it departs the queue. We identify models in this class by describing the input source (how jobs arrive), the demand jobs that it brings with them, the rules by which they compete for and receive service, and the capacity of the queue. Thus the elements of the queue includes the input process, service mechanism, queue discipline and output process.Queuing theory provides answers like mean waiting time in queue, mean response time, service time (service demand), user request arrival pattern and the distribution of users in the system, etc.
The Queuing system can be classified into 2 broad categories as Open or Closed System.
A system can be classified as Open or Closed based on certain characteristics it possesses.
Open System
These systems are usually internet facing where the request jobs arrives the system from outside externally and the arrival pattern of the requests are independent of the way the system processes the incoming requests. Thus the load on the system at any point of time is the external decision. The server workload for such systems can be represented as arrivals per unit time.
Closed System
These systems are usually client-server systems where the behavior of incoming request jobs are influenced by the server processing mechanism.The requests circulate continually forming a closed loop. Normally users fire a request, get a response from the server and think for certain amount of time (think time can be zero or non-zero value) and then again fire the request. Thus the load on the system at any point of time is decided by the system. The server workload for such systems can be represented as the number of users and think time.
The operational analysis is a set of basic quantitative relationships between performance quantities. The operational analysis is used to establish relationships among quantities based on measured or known data about computer systems.
Capacity planning is the process of determining what hardware and software configurations are required to adequately meet the application needs. It helps to define the hardware and network infrastructure needed to handle the expected and forecasted server load. Thus the goal of capacity planning is to provide satisfactory service levels to users in a cost-effective manner.
Capacity planning is carried out by measuring the number of requests the server currently processes and how much demand each request places on the server resources. Using this data, the resources (CPU, RAM, disk space, and network bandwidth) necessary to support current and future usage levels can be calculated.
The first and foremost reason is the user experience factor. Consider a system that can support only a given number of concurrent users while guaranteeing a reasonable response time. As many of us have experienced, when traffic increases on a major Web site that isn't adequately equipped to handle the surge, the response time deteriorates significantly. Am sure that many of us have experienced situations where sites, say payroll, when traffic increases (during last day of the month) on the site, the site isn’t adequately equipped to handle the sudden spike and hence resulting in response time deterioration. Studies have shown that if a site's response time is more than 10 seconds, then end users tend to leave. This is generally a bad thing and should be avoided, as there is no secret that a web site's downtime can result in a significant amount of loss to the business.
The second reason is that capacity planning helps you to decide how to allocate resources for a system in terms of the CPUs, RAM, Internet connection bandwidth, and LAN infrastructure needed to support the required performance levels and plan for future growth. Once we understand the limitations of the existing hardware configuration, we can estimate the amount of additional hardware needed to support any increased demands in performance.
Finally, capacity planning is important because it helps to answer the question of what hardware and software infrastructure is required to enable the current system to achieve expected performance objectives.
Software Performance Testing Handbook A Comprehensive Guide for Beginners
The Performance Test Reporting approach is totally different from the functional test reporting way. In the Performance Testing world, it is not easy to provide the pass/fail criteria as in the case of functional test report. The performance test report development is not as easy as everyone thinks because the test report should contain the symptoms and observations of the performance bottlenecks along with the recommendations. More than that, the information should be abstracted based on the audience of the test report.
Why is it Critical
The Performance Test Report is the document or presentation which contains the details of the performance test outcome. In case of business critical systems, the Performance Test Report would have a direct impact on the business decisions regarding the production release of the application. In some cases, the decisions related to infrastructure upgrades depend upon the test recommendations provided in the Performance Test Report. In case of products, few sections of the Test Report can also shared with all the clients as the performance benchmark. Though we should accept the fact the test report contents purely depends on the test objective, there are certain basic fundamental things which needs to be followed to develop a good test report.
How to report the results of performance testing
The foremost information to be provided in the Performance Test Report should be the overall objective of the performance testing. It should also provide the overview of the application and the business scenarios identified for testing and the user distribution model used for the testing. Though these details are provided in the test plan document, it is a good practice to include these information in the test report.
The main objective of the Performance Testing is to identify the server capacity (Throughput) to handle user load. The business might also be interested in the response time identification for the critical transactions. The Performance Testing activity provides the insight of SPEED, SCALABILITY and STABILITY of the application. The Performance Test Engineer should have this in mind while writing the test report.
The important information that should be available on the Performance Test Report includes the response time details in seconds, server throughput details in terms of transactions per second or Bytes per second , application errors or other errors encountered during the test and an indication about the hardware resource utilization levels.
Mere copying of few graphs from the performance testing tool doesn’t meet the objective of test report creation. The Performance Test Engineer should explain the test results through plain sentences in business English about what is the inference made from each of the graphs provided in the test report. In case of specific bottlenecks, symptoms needs to be explained as all the readers of the test report may not understand from looking at the graph. Each of the graphs should have its purpose detailed along with the clear details of x-axis and y-axis.
The Performance Test Engineer should ensure that right statistics are used and the report is customized to the intended audience. Also the performance test report is expected to have more visuals rather than providing the test results in text format.
Components of a good Test Report
A Performance test report will have different kind of readers like business people (management level people), technical members (architects, designers) and infra members (server admin and DBA). A good test report should meet the expectations of different kinds of readers and should make them understand the test results by adopting right level of abstraction.
It is always a best practice to have a separate section in the test report for providing the test results summary with a high level abstraction of information where technical details needs to be presented at high level but enough information on the test conclusions should be available as major business decisions are made based on this summary.
At a high-level, a performance test report should have the following information readily available.
• Performance Test objectives / goals (Service Level Agreements) • Test environment and the server configurations • Tests conducted (along with the deviations from the plan) • Performance Test Result summary • System Performance details with enough visuals • System Performance Bottlenecks details with enough visuals • Performance test conclusion and recommendations
Often Performance Test Engineers feel performance bottleneck analysis is a complex task. Unless you experience bottlenecks and experiment few things, one cannot become an expert in bottleneck analysis by reading articles or books. I agree that it is more of an art than science. But bottleneck analysis and isolation can be made easy when systematically approached and fundamental concepts in Queueing Theory are understood. This chapter explains few things which every Performance Test Engineer needs to have in mind during performance bottleneck analysis.
Scott’s Recommendation on Scatter Charts
Scatter Charts are one of the powerful tools for bottleneck analysis. It provides a quick view of the bottleneck and it is easy for a Performance Test Engineer to explain it to non-technical stakeholders.
As part of the bottleneck analysis, the first chart which every Performance Test Engineer needs to be look for is the response time chart. Look for the trend in the server processing time for each of the transaction or timer provided in the script during the span of the test. Always ignore the metrics collected during ramp up and ramp down as any conclusion should not be made while the server load is changing. Any metric that is collected during the stable constant load period can only be considered for the analysis.
A Scatter Chart can be created by having test time plotted in seconds on X-axis and the response time measure plotted in seconds on Y-axis. Each measurement represents the server response time for a specific transaction or timer provided in the script during the span of the test. A simple scatter chart is provided below. The blue dots represent the response time for loading the system home page and the red dots represent the response time for login transaction.
Performance Test Analysis
Though the performance testing activity is to simulate high loads on the system and to identify the system break points, practically not all the projects will have the objective to find bottlenecks on the system due to time or cost constraints. Many a time it so happens that project stakeholders expect the Performance Engineers to do just an application benchmarking (run the planned tests and report the system behavior), though there is a planned production move after the performance test. In my experience, many a time it happened that I end up convincing the project stakeholders about the business risk in doing application benchmarking without planning for bottleneck analysis and isolation. I have to accept that sometimes I have failed in my attempt and ended up doing benchmarking in spite of project being planned for production move.
It is very essential that every Performance Test Engineer should understand what is required for the application and suggest it to the stakeholders irrespective of limiting to time or cost constraints. However, acceptance is under stakeholder’s discretion.
Whenever any performance tests are run (benchmark tests, load tests, stress tests, etc), the first metric which needs to be looked at is the response time metric. This is the basic metric which needs to be checked to know the server’s processing time, time taken by the server to respond to user requests. Always adopt the practice of merging response time and running users graphs both for analysis and reporting purpose. Then look for the scatter chart pattern and start looking at the suspects.
The second important metric to look at is the server throughput, server’s processing capability in terms of requests handled per unit time or transaction handled per unit time or amount of bytes returned to the users per unit time. For this metric, there are several graphs available in the test tool like Hits per second, Transactions per second, Throughput (in Bytes/second).
I am sure you will not accept why I am talking about hits per second metric as this metric refers to the input load provided to the system and not the server’s output. To understand it better, consider this example. There is a small tank which has an inlet and outlet pipe of same capacity at the top and bottom of the tank respectively. Assume the tank doesn’t have any other leakage; Fill half of the tank by water having outlet pipe closed and then open the outlet pipe and observe the water inflow and outflow. We can observe that the rate of water flowing into the inlet pipe will be the same as the rate of water flowing out of the outlet pipe. This is true as long as the tank is in good stable condition. The same applies to the server during stable state. The incoming flow rate, A (arrivals per unit time) and out coming flow rate, C (completions per unit time) will be equal during the stable state (as long as there is no saturation in the server).
A = C where A is the arrival rate & C is the completion rate
The Hits per second graph provides the details on the requests handled by the server during unit time. For increasing loads, the hits per second should linearly increase and during the constant load period, the hits on the server should be constant. If there is a drop in the hits per second value, then it represents an issue on the server. This represents that the server is not able to handle the incoming requests which represents instability in the server. This issue needs to be cross verified with the server monitoring graphs to look for saturation issues of various service centers.
The errors observed during the tests needs to be analyzed and test data related issues needs to be isolated from the application errors. If test data related issues exist, it needs to be fixed and the test needs to be rerun to check for consistency in the issues.
Performance Testing Vs Profiling
The performance test tools have the capability to pinpoint the user actions which have high response time. By configuring right set of monitoring on the server infrastructure, hardware related bottlenecks could be effectively identified during performance tests. In the case of software bottlenecks, the performance test tools in general does not provide the details like which tier or software component consumes more processing time. Also, the performance test tools do not have the feature to drill down to the method call or component which contributes to high response time. Any issues related to the application code cannot be identified from the performance test tool.
Profiling is a dynamic program analysis which helps in understanding the program behavior by inserting specific code into the program code (instrumentation) to capture the call times, call stack, frequency of method calls, call performance, thread concurrency issues, memory usage limits, garbage collection details and heap behavior during the program execution. The profiling tools hooks into the JVM of the application (Java) and instruments the application classes to monitor its performance hence provide capabilities to show the call chains and helps in identifying the line of code which leads to high processing time for 1 user load. Profilers are considered as unit test tools as it helps in identifying code performance and it would be a best practice to do profiling and address the software issues before subjecting the system to performance tests.
HP Diagnostics tool facilitates drill down capabilities to identify the root cause for high response time transactions of the system. It helps to identify slow performing components, memory leaks, thread contentions, slow SQL query, slow layers and concurrency issues during the time of the load tests. The ability to provide the server performance behavior during the loaded condition makes HP Diagnostics tool special and different from other profiler tools like Dev Partner, Glow code, etc. The Diagnostics tool can be integrated with the HP performance test tools (Load Runner, PerformanceCenter), the same performance test scripts used for performance testing could be used to create the required load on the server and the server performance characteristics could be monitored and diagnosed.
Performance monitoring is the process of collecting and analyzing the server data to compare the server statistics against the expected values. It helps the performance tester to have the health check of the server during the test. By monitoring the servers during the test, one can identify the server behavior for load condition and take steps to change the server behavior by adopting software or hardware performance tuning activities.
Each performance counter helps in identifying a specific value about the server performance. For example, % CPU Utilization is a performance counter which helps in identifying the utilization level of the CPU.
In a nutshell, server monitoring should provide information on four parameters of any system: Latency, Throughput, Utilization and Efficiency, which helps in answering the following questions.
Is your server available?
How busy is your CPU?
Is there enough Primary Memory (RAM)?
Is the disk fast enough?
Is there any other hardware issues?
Is the hardware issue result of software malfunctioning?
Always start with few counters and once you notice a specific problem, start adding few more counters related to the symptom. Start monitoring the performance of the major resources like CPU, Memory, Disk or Network. This section provides you the details about key counters from each of above mentioned 4 areas which are very important for a Performance Tester to know.
Processor Bottlenecks
The bottlenecks related to processor (CPU) are comparatively easy to identify. The important performance counters that helps in identifying the processor bottleneck includes
% Processor Utilization (Processor_Total: % Processor Time) – This counter helps in knowing how busy the system is. It indicates the processor activity. It is the average percentage of elapsed time that the processor spends to execute a productive (non-idle) thread. A consistent level of more than 80% utilization (in case of single CPU machines) indicates that there is not enough CPU capacity. It is worth further investigation using other processor counters.
% User time (Processor_Total: % User Time) – This refers to the processor’s time spent in handling the application related processes. A high percentage indicates that the application is consuming high CPU. The process level counters needs to be monitored to understand which user process consumes more CPU.
% Privileged time (Processor_Total: % Privilege Time) – This refers to the processor’s time spent in handling the kernel mode processes. A high value indicates that the processor is too busy in handling other operating system related activities. It needs immediate attention from the system administrator to check the system configuration or service.
% I/O Wait (%wio - in case of UNIX platforms) – This refers to the percentage wait for completion of I/O activity. It is a good indication to confirm whether the threads are waiting for the I/O completion.
Processor Queue Length (System: Processor Queue Length) – This counter helps in identifying how many threads are waiting in queue for execution. A consistent queue length of more than 2 indicates bottleneck and it is worth investigation. Generally if the queue length is more than the number of CPUs available in the system, then it might reduce the system performance. A high value of % usr time coupled with high processor queue length indicates the processor bottleneck.
Other counters of interest:
Other counters like Processor: Interrupts per second, System: Context Switches per second can be used in case of any specific issues. Interrupts per second refers to the number of interrupts that the hardware devices sends to the processor. A consistent value of above 1000 in Interrupts per second indicates hardware failure or driver configuration issues. Context Switches refers to the switching of the processor from a lower priority thread to a high priority thread. A consistent value of above 15000 per second per processor indicates the presence of too many threads of same priority and possibility of having blocked threads.
Windows operating system comes with a performance monitoring tool called Perfmon. This monitoring tool can be used to collect the server statistics during the performance test run. Normally, most of the performance testing tools have its own monitors to monitor the system resources of the system under test. In this case, it becomes easy to compare the system load related metrics with the system resource utilization metrics to arrive at a conclusion. Infact, any performance testing tool that monitors windows machine internally talks to perfmon to collect the system resource utilization details.
There are lots of licensed tools available in the market which is used for post production monitoring. These tools are used to capture the web server traffic details and provide online traffic details. A very popular tool of this category is WebTrends. Many organizations uses this tool to get to the traffic trends of an application running in production environment. There are other tools like HP OpenView tools which run in production servers and monitor the server resource utilization levels. It provides alarming mechanism to indicate the heavy usage and provides easy bottleneck isolation capabilities. But due to the cost involved with these kinds of tools, most of small organizations don’t opt for them. But post production monitoring data would be of great use for designing realistic performance tests.
·Allows you to analyze and isolate the performance problem.
·Understand the resource utilization of the server and make best use of them.
·Plan for Capacity Planning activity based on the resource utilization level.
·Provides the server performance details offline (by creating a alert of sending a mail/message when resource utilization reaches the threshold value).
