Chapter 12: Road Ahead – Moving towards advanced Performance Engineering techniques

Software Performance Testing Handbook

A Comprehensive Guide for Beginners

Moving towards Performance Engineering

As discussed in earlier chapters, mere plans for Performance testing during the end of the SDLC cycle is considered as a reactive approach wherein Performance Engineering approach aims at integrating the performance engineering activities from starting of the SDLC to build a system with high performance. Thus, Performance testing forms a subset of the Performance Engineering activity. Accordingly, the competencies required for a performance engineer is little different from the performance tester.

Software Performance Testing is the act of evaluating the software system for its performance and finding the bottlenecks in the system. Software Performance Engineering is the systematic approach of constructing the software that meets the performance objectives.

Nowadays, IT companies have slowly started adopting the performance engineering activities for critical applications though 70-80% of them have adopted mere performance testing activities.

What is Queuing Theory?

Queuing Theory is the mathematical study of waiting lines (or queues). It is interchangeably spelled as Queueing Theory or Queuing Theory. It is considered as a branch of Probability Theory and Operational Research as it is used in making business decisions. It is used in various fields like telecommunications, networks, traffic management, etc. The notation for describing the queuing model was first introduced by David .G. Kendal in 1953.

The figure represents a model of a single queue. Jobs arrive at the queue from the input source, each job bringing a demand for service from the queue. If the queue becomes full, the arriving job is not allowed to join the queue. If the job enters the queue, it may compete with other jobs for service from the queue. When a job completes its service, it departs the queue. We identify models in this class by describing the input source (how jobs arrive), the demand jobs that it brings with them, the rules by which they compete for and receive service, and the capacity of the queue. Thus the elements of the queue includes the input process, service mechanism, queue discipline and output process.Queuing theory provides answers like mean waiting time in queue, mean response time, service time (service demand), user request arrival pattern and the distribution of users in the system, etc.

Types of Queuing System

The Queuing system can be classified into 2 broad categories as Open or Closed System.

A system can be classified as Open or Closed based on certain characteristics it possesses.

Open System

These systems are usually internet facing where the request jobs arrives the system from outside externally and the arrival pattern of the requests are independent of the way the system processes the incoming requests. Thus the load on the system at any point of time is the external decision. The server workload for such systems can be represented as arrivals per unit time.

Closed System

These systems are usually client-server systems where the behavior of incoming request jobs are influenced by the server processing mechanism. The requests circulate continually forming a closed loop. Normally users fire a request, get a response from the server and think for certain amount of time (think time can be zero or non-zero value) and then again fire the request. Thus the load on the system at any point of time is decided by the system. The server workload for such systems can be represented as the number of users and think time.

Operational Laws

The operational analysis is a set of basic quantitative relationships between performance quantities. The operational analysis is used to establish relationships among quantities based on measured or known data about computer systems.

Capacity Planning

Capacity planning is the process of determining what hardware and software configurations are required to adequately meet the application needs. It helps to define the hardware and network infrastructure needed to handle the expected and forecasted server load. Thus the goal of capacity planning is to provide satisfactory service levels to users in a cost-effective manner.

Capacity planning is carried out by measuring the number of requests the server currently processes and how much demand each request places on the server resources. Using this data, the resources (CPU, RAM, disk space, and network bandwidth) necessary to support current and future usage levels can be calculated.

Why Capacity Planning is required

The first and foremost reason is the user experience factor. Consider a system that can support only a given number of concurrent users while guaranteeing a reasonable response time. As many of us have experienced, when traffic increases on a major Web site that isn't adequately equipped to handle the surge, the response time deteriorates significantly. Am sure that many of us have experienced situations where sites, say payroll, when traffic increases (during last day of the month) on the site, the site isn’t adequately equipped to handle the sudden spike and hence resulting in response time deterioration. Studies have shown that if a site's response time is more than 10 seconds, then end users tend to leave. This is generally a bad thing and should be avoided, as there is no secret that a web site's downtime can result in a significant amount of loss to the business.

The second reason is that capacity planning helps you to decide how to allocate resources for a system in terms of the CPUs, RAM, Internet connection bandwidth, and LAN infrastructure needed to support the required performance levels and plan for future growth. Once we understand the limitations of the existing hardware configuration, we can estimate the amount of additional hardware needed to support any increased demands in performance.

Finally, capacity planning is important because it helps to answer the question of what hardware and software infrastructure is required to enable the current system to achieve expected performance objectives.

Chapter 11: Performance Test Reporting

Software Performance Testing Handbook
A Comprehensive Guide for Beginners

The Performance Test Reporting approach is totally different from the functional test reporting way. In the Performance Testing world, it is not easy to provide the pass/fail criteria as in the case of functional test report. The performance test report development is not as easy as everyone thinks because the test report should contain the symptoms and observations of the performance bottlenecks along with the recommendations. More than that, the information should be abstracted based on the audience of the test report.


Why is it Critical

The Performance Test Report is the document or presentation which contains the details of the performance test outcome. In case of business critical systems, the Performance Test Report would have a direct impact on the business decisions regarding the production release of the application. In some cases, the decisions related to infrastructure upgrades depend upon the test recommendations provided in the Performance Test Report. In case of products, few sections of the Test Report can also shared with all the clients as the performance benchmark. Though we should accept the fact the test report contents purely depends on the test objective, there are certain basic fundamental things which needs to be followed to develop a good test report.

How to report the results of performance testing

The foremost information to be provided in the Performance Test Report should be the overall objective of the performance testing. It should also provide the overview of the application and the business scenarios identified for testing and the user distribution model used for the testing. Though these details are provided in the test plan document, it is a good practice to include these information in the test report.

The main objective of the Performance Testing is to identify the server capacity (Throughput) to handle user load. The business might also be interested in the response time identification for the critical transactions. The Performance Testing activity provides the insight of SPEED, SCALABILITY and STABILITY of the application. The Performance Test Engineer should have this in mind while writing the test report.

The important information that should be available on the Performance Test Report includes the response time details in seconds, server throughput details in terms of transactions per second or Bytes per second , application errors or other errors encountered during the test and an indication about the hardware resource utilization levels.

Mere copying of few graphs from the performance testing tool doesn’t meet the objective of test report creation. The Performance Test Engineer should explain the test results through plain sentences in business English about what is the inference made from each of the graphs provided in the test report. In case of specific bottlenecks, symptoms needs to be explained as all the readers of the test report may not understand from looking at the graph. Each of the graphs should have its purpose detailed along with the clear details of x-axis and y-axis.

The Performance Test Engineer should ensure that right statistics are used and the report is customized to the intended audience. Also the performance test report is expected to have more visuals rather than providing the test results in text format.

Components of a good Test Report

A Performance test report will have different kind of readers like business people (management level people), technical members (architects, designers) and infra members (server admin and DBA). A good test report should meet the expectations of different kinds of readers and should make them understand the test results by adopting right level of abstraction.

It is always a best practice to have a separate section in the test report for providing the test results summary with a high level abstraction of information where technical details needs to be presented at high level but enough information on the test conclusions should be available as major business decisions are made based on this summary.

At a high-level, a performance test report should have the following information readily available.

• Performance Test objectives / goals (Service Level Agreements)
• Test environment and the server configurations
• Tests conducted (along with the deviations from the plan)
• Performance Test Result summary
• System Performance details with enough visuals
• System Performance Bottlenecks details with enough visuals
• Performance test conclusion and recommendations

Chapter 10: Performance Bottleneck Analysis

Software Performance Testing Handbook

A Comprehensive Guide for Beginners

Often Performance Test Engineers feel performance bottleneck analysis is a complex task. Unless you experience bottlenecks and experiment few things, one cannot become an expert in bottleneck analysis by reading articles or books. I agree that it is more of an art than science. But bottleneck analysis and isolation can be made easy when systematically approached and fundamental concepts in Queueing Theory are understood. This chapter explains few things which every Performance Test Engineer needs to have in mind during performance bottleneck analysis.

Scott’s Recommendation on Scatter Charts

Scatter Charts are one of the powerful tools for bottleneck analysis. It provides a quick view of the bottleneck and it is easy for a Performance Test Engineer to explain it to non-technical stakeholders.

As part of the bottleneck analysis, the first chart which every Performance Test Engineer needs to be look for is the response time chart. Look for the trend in the server processing time for each of the transaction or timer provided in the script during the span of the test. Always ignore the metrics collected during ramp up and ramp down as any conclusion should not be made while the server load is changing. Any metric that is collected during the stable constant load period can only be considered for the analysis.

A Scatter Chart can be created by having test time plotted in seconds on X-axis and the response time measure plotted in seconds on Y-axis. Each measurement represents the server response time for a specific transaction or timer provided in the script during the span of the test. A simple scatter chart is provided below. The blue dots represent the response time for loading the system home page and the red dots represent the response time for login transaction.

Performance Test Analysis

Though the performance testing activity is to simulate high loads on the system and to identify the system break points, practically not all the projects will have the objective to find bottlenecks on the system due to time or cost constraints. Many a time it so happens that project stakeholders expect the Performance Engineers to do just an application benchmarking (run the planned tests and report the system behavior), though there is a planned production move after the performance test. In my experience, many a time it happened that I end up convincing the project stakeholders about the business risk in doing application benchmarking without planning for bottleneck analysis and isolation. I have to accept that sometimes I have failed in my attempt and ended up doing benchmarking in spite of project being planned for production move.

It is very essential that every Performance Test Engineer should understand what is required for the application and suggest it to the stakeholders irrespective of limiting to time or cost constraints. However, acceptance is under stakeholder’s discretion.

Whenever any performance tests are run (benchmark tests, load tests, stress tests, etc), the first metric which needs to be looked at is the response time metric. This is the basic metric which needs to be checked to know the server’s processing time, time taken by the server to respond to user requests. Always adopt the practice of merging response time and running users graphs both for analysis and reporting purpose. Then look for the scatter chart pattern and start looking at the suspects.

The second important metric to look at is the server throughput, server’s processing capability in terms of requests handled per unit time or transaction handled per unit time or amount of bytes returned to the users per unit time. For this metric, there are several graphs available in the test tool like Hits per second, Transactions per second, Throughput (in Bytes/second).

I am sure you will not accept why I am talking about hits per second metric as this metric refers to the input load provided to the system and not the server’s output. To understand it better, consider this example. There is a small tank which has an inlet and outlet pipe of same capacity at the top and bottom of the tank respectively. Assume the tank doesn’t have any other leakage; Fill half of the tank by water having outlet pipe closed and then open the outlet pipe and observe the water inflow and outflow. We can observe that the rate of water flowing into the inlet pipe will be the same as the rate of water flowing out of the outlet pipe. This is true as long as the tank is in good stable condition. The same applies to the server during stable state. The incoming flow rate, A (arrivals per unit time) and out coming flow rate, C (completions per unit time) will be equal during the stable state (as long as there is no saturation in the server).

A = C where A is the arrival rate & C is the completion rate

The Hits per second graph provides the details on the requests handled by the server during unit time. For increasing loads, the hits per second should linearly increase and during the constant load period, the hits on the server should be constant. If there is a drop in the hits per second value, then it represents an issue on the server. This represents that the server is not able to handle the incoming requests which represents instability in the server. This issue needs to be cross verified with the server monitoring graphs to look for saturation issues of various service centers.

The errors observed during the tests needs to be analyzed and test data related issues needs to be isolated from the application errors. If test data related issues exist, it needs to be fixed and the test needs to be rerun to check for consistency in the issues.

Performance Testing Vs Profiling

The performance test tools have the capability to pinpoint the user actions which have high response time. By configuring right set of monitoring on the server infrastructure, hardware related bottlenecks could be effectively identified during performance tests. In the case of software bottlenecks, the performance test tools in general does not provide the details like which tier or software component consumes more processing time. Also, the performance test tools do not have the feature to drill down to the method call or component which contributes to high response time. Any issues related to the application code cannot be identified from the performance test tool.

Profiling is a dynamic program analysis which helps in understanding the program behavior by inserting specific code into the program code (instrumentation) to capture the call times, call stack, frequency of method calls, call performance, thread concurrency issues, memory usage limits, garbage collection details and heap behavior during the program execution. The profiling tools hooks into the JVM of the application (Java) and instruments the application classes to monitor its performance hence provide capabilities to show the call chains and helps in identifying the line of code which leads to high processing time for 1 user load. Profilers are considered as unit test tools as it helps in identifying code performance and it would be a best practice to do profiling and address the software issues before subjecting the system to performance tests.

HP Diagnostics

HP Diagnostics tool facilitates drill down capabilities to identify the root cause for high response time transactions of the system. It helps to identify slow performing components, memory leaks, thread contentions, slow SQL query, slow layers and concurrency issues during the time of the load tests. The ability to provide the server performance behavior during the loaded condition makes HP Diagnostics tool special and different from other profiler tools like Dev Partner, Glow code, etc. The Diagnostics tool can be integrated with the HP performance test tools (Load Runner, Performance Center), the same performance test scripts used for performance testing could be used to create the required load on the server and the server performance characteristics could be monitored and diagnosed.

Chapter 9: Performance Test Monitoring

Software Performance Testing Handbook

A Comprehensive Guide for Beginners

Introduction to Performance Monitoring

Performance monitoring is the process of collecting and analyzing the server data to compare the server statistics against the expected values. It helps the performance tester to have the health check of the server during the test. By monitoring the servers during the test, one can identify the server behavior for load condition and take steps to change the server behavior by adopting software or hardware performance tuning activities.

Each performance counter helps in identifying a specific value about the server performance. For example, % CPU Utilization is a performance counter which helps in identifying the utilization level of the CPU.

In a nutshell, server monitoring should provide information on four parameters of any system: Latency, Throughput, Utilization and Efficiency, which helps in answering the following questions.

• Is your server available?
• How busy is your CPU?
• Is there enough Primary Memory (RAM)?
• Is the disk fast enough?
• Is there any other hardware issues?
• Is the hardware issue result of software malfunctioning?

Key Performance Counters

Always start with few counters and once you notice a specific problem, start adding few more counters related to the symptom. Start monitoring the performance of the major resources like CPU, Memory, Disk or Network. This section provides you the details about key counters from each of above mentioned 4 areas which are very important for a Performance Tester to know.

Processor Bottlenecks

The bottlenecks related to processor (CPU) are comparatively easy to identify. The important performance counters that helps in identifying the processor bottleneck includes

% Processor Utilization (Processor_Total: % Processor Time) – This counter helps in knowing how busy the system is. It indicates the processor activity. It is the average percentage of elapsed time that the processor spends to execute a productive (non-idle) thread. A consistent level of more than 80% utilization (in case of single CPU machines) indicates that there is not enough CPU capacity. It is worth further investigation using other processor counters.

% User time (Processor_Total: % User Time) – This refers to the processor’s time spent in handling the application related processes. A high percentage indicates that the application is consuming high CPU. The process level counters needs to be monitored to understand which user process consumes more CPU.

% Privileged time (Processor_Total: % Privilege Time) – This refers to the processor’s time spent in handling the kernel mode processes. A high value indicates that the processor is too busy in handling other operating system related activities. It needs immediate attention from the system administrator to check the system configuration or service.

% I/O Wait (%wio - in case of UNIX platforms) – This refers to the percentage wait for completion of I/O activity. It is a good indication to confirm whether the threads are waiting for the I/O completion.

Processor Queue Length (System: Processor Queue Length) – This counter helps in identifying how many threads are waiting in queue for execution. A consistent queue length of more than 2 indicates bottleneck and it is worth investigation. Generally if the queue length is more than the number of CPUs available in the system, then it might reduce the system performance. A high value of % usr time coupled with high processor queue length indicates the processor bottleneck.

Other counters of interest:

Other counters like Processor: Interrupts per second, System: Context Switches per second can be used in case of any specific issues. Interrupts per second refers to the number of interrupts that the hardware devices sends to the processor. A consistent value of above 1000 in Interrupts per second indicates hardware failure or driver configuration issues. Context Switches refers to the switching of the processor from a lower priority thread to a high priority thread. A consistent value of above 15000 per second per processor indicates the presence of too many threads of same priority and possibility of having blocked threads.

Performance Monitoring in Windows platform

Windows operating system comes with a performance monitoring tool called Perfmon. This monitoring tool can be used to collect the server statistics during the performance test run. Normally, most of the performance testing tools have its own monitors to monitor the system resources of the system under test. In this case, it becomes easy to compare the system load related metrics with the system resource utilization metrics to arrive at a conclusion. Infact, any performance testing tool that monitors windows machine internally talks to perfmon to collect the system resource utilization details.

Post production Monitoring

There are lots of licensed tools available in the market which is used for post production monitoring. These tools are used to capture the web server traffic details and provide online traffic details. A very popular tool of this category is WebTrends. Many organizations uses this tool to get to the traffic trends of an application running in production environment. There are other tools like HP OpenView tools which run in production servers and monitor the server resource utilization levels. It provides alarming mechanism to indicate the heavy usage and provides easy bottleneck isolation capabilities. But due to the cost involved with these kinds of tools, most of small organizations don’t opt for them. But post production monitoring data would be of great use for designing realistic performance tests.

Benefits of Performance Monitoring

· Allows you to analyze and isolate the performance problem.

· Understand the resource utilization of the server and make best use of them.

· Plan for Capacity Planning activity based on the resource utilization level.

· Provides the server performance details offline (by creating a alert of sending a mail/message when resource utilization reaches the threshold value).

Chapter 8: Performance Test Execution

Software Performance Testing Handbook
A Comprehensive Guide for Beginners

In the Internet, the competitors are only a mouse click away. Hence the web sites should provide high performing, time effective application thereby increasing the satisfaction level of the end users. Dissatisfied users would abandon the site and never come back. Bad mouthing would spoil other user’s interest towards the web site. Hence there is a high necessity to design applications with high performance to meet the expectations of the end user.

There are certain activities that need to be performed before we plan for the performance tests for a web site. By ensuring these activities, we could improve the accuracy of the performance tests and would run tests simulating the real time usage.

Quality of Service of web application


The QOS (Quality Of Service) of a web application is normally measured using the system response time and system throughput. It is about ‘How fast is the system able to respond to the customer requests?’ and ‘How much load the server can withstand by providing quick response to the customers?’ There is a high chance for losing the business when the customers are not satisfied with the QOS of the web site.

Providing QOS to the end users of the system is a big challenge faced by the application designers. The objective of the system testing is to identify the functional defects before the end user finds it. The system needs to be assured not only for its functional stability but also for its availability and responsiveness at all times of the user load. The end users will not be happy just by having a functionally stable system; the system should be very responsive and available enough at all times of user accesses.

It is very important for a web application to respond quickly with minimal processing time irrespective of number of users accessing the system. Though the application might be functionally stable for 1 user, but there could be inherent design issues or code level issues which might create show stopper issues during multi user load. Hence it is highly required to test the system performance before releasing the system to the open community.

Types of Performance Tests

There are various types of tests that could be performed on an application during the performance certification activity. The objective and importance of various types of tests are discussed below.
1. Baseline test
2. Benchmark test
3. Load test
4. Stress test
5. Soak test
6. Endurance or Longevity test

But it is not required that all application needs to undergo all of these tests during performance testing. It purely depends on the application context and its usage pattern. It is always recommended to carry out any special tests which are different from those mentioned above, if the application usage pattern demands one.
Performance Test Execution Approach

Let us recollect the basic objective of running a performance test. The Performance test is conducted to identify and certify how fast the application responds and at what point of load the application performance starts degrading. The very important expectation on the performance test activity is the ability to run more realistic tests. By injecting the required number of users load during the test, realistic end user usage pattern cannot be achieved. The server workload needs to be analyzed which is purely based on the expected accuracy level in the test results. For a performance critical application, often high accuracy is expected combined with high level of confidence about the system performance.

Normally, for performance testing, paretto’s 80-20 rule should be followed. After identifying the most frequently used business flows, user distribution among the identified business flows needs to be done carefully. Though 100% real time simulation is impossible, lot of analysis should go into this analysis to come up with more accurate realistic numbers. Usually for this analysis, web log analysis provides lot of inputs in case of applications where usage history exists. A high accuracy level can be expected in case of the availability of real time usage data. For applications, going to production for the first time, expected accuracy level should be less due to the non-availability of real time data.
It is always better to neglect those business flows which are seen or expected to be occasionally used during peak business hours. Rather during performance tests, subject the server to a constant load for 15-30 minutes approximately (also depends on application usage pattern) to get more samples of server behavior (with respect to response time and resource utilization levels) during loaded condition and then certify the application for the subjected load. A test could be called realistic only when the user load and the transaction mix are appropriate along with the associated confidence about the server performance behavior.

It is required to verify the system performance for more number of samples and then conclude the performance metrics for response time, server utilization levels, etc. The test should always have sufficient time for user ramp up, before subjecting the system to the expected load period. All the measurements about the server performance need to be taken only during the stable load period. The test should be configured in such a way that all users are actively accessing the system during the stable load period. If the peak expected server load is for example 200 requests per second, then the test should be scheduled in such a way that that running users should create a constant load of 200 requests per second atleast for 10 to 15 minutes duration. Though the peak expected user load (200 requests per second) occurs for less number of times in the real time with sufficient time interval between the spikes, the performance test should subject the system for a constant load to get more samples on the server performance. This helps in assuring the server performance behavior for a peak traffic situations. Hence the Performance Test Engineer should come up with a proper test strategy in order to test the system for realistic load and also at the same time, the test should be able to guarantee the system performance behavior for the peak load situations.

Let us discuss an example to validate the load test scenario created for the business requirements. The application needs to be validated for the 100 user load which consists of following 3 business scenarios.

Business Scenario A -60% of target load
Business Scenario B -30% of target load
Business Scenario C -10% of target load

Based on the inputs from business analysis, it was found that scenario A will be executed 6 times in an hour, scenario B will be executed 2 times in an hour and scenario C will be executed 1 time in an hour. Also, it was known that the peak server load was 200 requests per second during the peak hour.

The following load test is scheduled in order to meet the above test requirements.


This load test strategy meets the user load targets (100 users accessing the system for 40 minutes of stable load period) and the number of executions of each business scenarios (6 iterations of A scenario, 2 iterations of B scenario and 1 iteration of C scenario). During the stable load period, when all three business scenarios were running (40:00 to 50:00 mm:ss), the server load created on the server was about 200 requests per second. Hence the test seems to be realistic to the production load.

The above test scenario meets the performance test requirements of the applications and it provides the system performance figures for the realistic system usage. But what is missing in the above load test scenario is the confidence level of server being able to handle the 200 requests per second, as there was only 1 sample collected during the interval (40:00 to 50:00 mm:ss) when all three business scenarios were running. A load test should aim for assessing the server performance for realistic usage pattern and also create a high confidence by analyzing the system performance for atleast 3 to 4 samples. The confidence level increases only when more number of samples is made available and analyzed for server behavior. Also, the server resource utilization levels needs to be assured only by subjecting the system to a stable constant load. Hence it is recommended to run all the three business scenarios together for atleast 20 to 30 minutes to collect enough samples on the response time metrics and server resource utilization metrics by maintaining the load at 200 requests per second. This can be combined in the same load test or can be run as a separate test depending the application performance requirements and performance behavior of the system.

The moment we talk about realistic tests, think time should be the high priority one worth spending time for detailed analysis. In spite of having correct business flows and user distributions between the business flows, the moment think time is not realistic, the entire test goes for a toss leading to unrealistic tests. The think time decides the user request arrival pattern of an application which decides the load on the server at any point of time. Usually it is always recommended to use realistic think time between transactions and randomize it while running the test (say 80% to 120% of configured think time). This randomization helps is simulating the think time of inexperienced user of the web site and the frequent user of the web site.

Chapter 7: Application Benchmarking

Software Performance Testing Handbook

A Comprehensive Guide for Beginners

What is benchmarking?

In general, Benchmarking is the process of determining who is very best, who sets the standard, and what that standard is. When it comes to the Performance Testing context, most of the time we often use the word benchmark. Benchmarking is the process of the determining the relative performance of the software program by running the standard set of tests. We can benchmark the hardware or software performance by comparing its relative performance figures. For example, we can benchmark the software application across various application servers by deploying it in Websphere server, Weblogic server and JBoss server to compare the performance of the application in each server. We can benchmark the hardware requirement for the software by running the tests on Dell server, HP Proliant server and IBM servers to compare the CPU, Disk and Memory utilization details.

Why should we benchmark the applications?

Unless we compare ourselves with others against a common measure (for example say height) which is a measure applicable for both entities, we cannot say who is best. It becomes very tough if we don’t know what is the standard measure that can be used to compare oneself against the other. The same is applicable for the software systems. One always needs to know where their competitors stand. In order to compare different software or hardware performance, we need to have the common standard measurements.

Industry Standards for Benchmarking


It becomes impossible for the application owners to test the application performance against variety of the server machines available in the market and to choose among them due to cost factor. There are organizations available in the market which does this benchmarking. The application owners could refer to these industrial benchmarks to decide on their infrastructure requirements. Transactions Processing Performance Council (TPC), Standard Performance Evaluation Council (SPEC), Synchromesh benchmarks, etc are the industry standard benchmarks available. There are other open source and vendor specific benchmarks available in the market. These organizations perform the testing on different servers with varied hardware configurations and provide the performance figures. As we all know we need to have some common measure to compare with the competitors, these industry standards provide the list of measures which is common across server platforms.


Transaction processing Performance Council (TPC)

The TPC is a non-profit corporation founded in 1980’s to define transaction processing and database benchmarks and to disseminate objective, verifiable TPC performance data to the industry. The TPC benchmarks are widely used today in evaluating the performance of computer systems.

The TPC benchmarks involve the measurement and evaluation of computer functions and operations through transaction as it is commonly understood in the business world. A typical transaction, as defined by the TPC, would include updating to a database system, a set of operations including disk read/writes, operating system calls, or some form of data transfer from one subsystem to another. There are different types of benchmarks available in TPC. It includes TPC-App, TPC-C, TPC-E and TPC-H. The following information is taken from the TPC site - http://www.tpc.org/.

Standard Performance Evaluation Council

The SPEC is a non-profit organization that aims to produce fair, impartial and meaningful benchmarks for the computers. SPEC was founded in 1988 and the goal is to ensure that the marketplace has a fair and useful set of metrics to differentiate candidate systems. Its member organizations include leading computer and software manufacturers. SPEC benchmarks are widely used today in evaluating the performance of computer systems; the results are published on the SPEC web site - http://www.spec.org/ .

Mistakes in Benchmarking

One needs to understand the fact that benchmarks provided by organizations like TPC, SPEC are based on the workload created by a typical application used for the performance testing. If the application owners develop applications with a specific workload which is totally different from the typical application workload used for benchmarking, then it is not an apple to apple comparison. But we don’t have any choice as it is impossible to test our product across different configurations to choose the best one. Hence we can refer to the benchmarks and add appropriate risk buffer if the workload of the application under test is completely different from the one used for benchmarking.