Most of the market available performance testing tools available for web services performance testing does not provide support to WCF (Windows Communication Foundation) web services performance testing.
The following are the tools which supports WCF web servcies performance testing.
> WCFStorm
> WCF Load Test
> SOAtest
SOAPUI doesnt support WCF web services performance testing. Out of above 3 tools, we have finlized on the SOAtest 5.5 tool to carryout WCF web services performance testing.
Monday, April 20, 2009
Web Services Performance Testing
Web Services Performance Testing using SOAPUI Recently i had a chance to recommend a performance testing tool to carryout performance testing for web services system. I evaluated a series of open source and licensed tools and finally concluded SOAPUI would be meet the requirement for our system. soapUI provides extensive load testing functionality allowing you to do the following: Behavioral LoadTesting : analyze performance behaviour under varying load with different load strategies. Performance LoadTesting : find maximal performance available using thread strategies and Command Line LoadTest execution. Requirements Driven LoadTesting : define performance requirements and continuously validate using Load Test assertions. SOAPUI helps in the following.
Functional LoadTesting : validate functionality under load using standard TestCase methods.
> Validate a Web Services performance under different Load scenarios.
> Maintain Functional validations to see that they don't "break" under load.
> Run several load tests simultaneously to see how they affect each other.
Its a freeware tool which best suits web services load testing. Its commercial version is called SOAPUI Pro includes productivity enahancements. In Overall SOAPUI meets most the requirements for a Web services performance testing and it is a apt tool to meet most of the web services.
Monday, March 30, 2009
Software Performance Testing Handbook - A Comprehensive Guide for Begineers
Dear viewers,
Hereby am publishing my first version of ebook on performance testing which was pending for a long time :). I have tried my best to compile my learnings and experience to help you out. I guess you will get benefited out of it.
Happy Learning!!!
You can download the book through the online academy @ elitesouls.in :
http://elitesouls.teachable.com/p/performance-testing-foundational-learning/?
Hereby am publishing my first version of ebook on performance testing which was pending for a long time :). I have tried my best to compile my learnings and experience to help you out. I guess you will get benefited out of it.
Happy Learning!!!
You can download the book through the online academy @ elitesouls.in :
http://elitesouls.teachable.com/p/performance-testing-foundational-learning/?
Tuesday, December 2, 2008
Chapter 12: Road Ahead – Moving towards advanced Performance Engineering techniques
Software Performance Testing Handbook
A Comprehensive Guide for Beginners
Moving towards Performance Engineering
As discussed in earlier chapters, mere plans for Performance testing during the end of the SDLC cycle is considered as a reactive approach wherein Performance Engineering approach aims at integrating the performance engineering activities from starting of the SDLC to build a system with high performance. Thus, Performance testing forms a subset of the Performance Engineering activity. Accordingly, the competencies required for a performance engineer is little different from the performance tester.
Software Performance Testing is the act of evaluating the software system for its performance and finding the bottlenecks in the system. Software Performance Engineering is the systematic approach of constructing the software that meets the performance objectives.
Nowadays, IT companies have slowly started adopting the performance engineering activities for critical applications though 70-80% of them have adopted mere performance testing activities.
Queuing Theory is the mathematical study of waiting lines (or queues). It is interchangeably spelled as Queueing Theory or Queuing Theory. It is considered as a branch of Probability Theory and Operational Research as it is used in making business decisions. It is used in various fields like telecommunications, networks, traffic management, etc. The notation for describing the queuing model was first introduced by David .G. Kendal in 1953.
The figure represents a model of a single queue. Jobs arrive at the queue from the input source, each job bringing a demand for service from the queue. If the queue becomes full, the arriving job is not allowed to join the queue. If the job enters the queue, it may compete with other jobs for service from the queue. When a job completes its service, it departs the queue. We identify models in this class by describing the input source (how jobs arrive), the demand jobs that it brings with them, the rules by which they compete for and receive service, and the capacity of the queue. Thus the elements of the queue includes the input process, service mechanism, queue discipline and output process.Queuing theory provides answers like mean waiting time in queue, mean response time, service time (service demand), user request arrival pattern and the distribution of users in the system, etc.
The Queuing system can be classified into 2 broad categories as Open or Closed System.
A system can be classified as Open or Closed based on certain characteristics it possesses.
Open System
These systems are usually internet facing where the request jobs arrives the system from outside externally and the arrival pattern of the requests are independent of the way the system processes the incoming requests. Thus the load on the system at any point of time is the external decision. The server workload for such systems can be represented as arrivals per unit time.
Closed System
These systems are usually client-server systems where the behavior of incoming request jobs are influenced by the server processing mechanism. The requests circulate continually forming a closed loop. Normally users fire a request, get a response from the server and think for certain amount of time (think time can be zero or non-zero value) and then again fire the request. Thus the load on the system at any point of time is decided by the system. The server workload for such systems can be represented as the number of users and think time.
The operational analysis is a set of basic quantitative relationships between performance quantities. The operational analysis is used to establish relationships among quantities based on measured or known data about computer systems.
Capacity planning is the process of determining what hardware and software configurations are required to adequately meet the application needs. It helps to define the hardware and network infrastructure needed to handle the expected and forecasted server load. Thus the goal of capacity planning is to provide satisfactory service levels to users in a cost-effective manner.
Capacity planning is carried out by measuring the number of requests the server currently processes and how much demand each request places on the server resources. Using this data, the resources (CPU, RAM, disk space, and network bandwidth) necessary to support current and future usage levels can be calculated.
Why Capacity Planning is required
The first and foremost reason is the user experience factor. Consider a system that can support only a given number of concurrent users while guaranteeing a reasonable response time. As many of us have experienced, when traffic increases on a major Web site that isn't adequately equipped to handle the surge, the response time deteriorates significantly. Am sure that many of us have experienced situations where sites, say payroll, when traffic increases (during last day of the month) on the site, the site isn’t adequately equipped to handle the sudden spike and hence resulting in response time deterioration. Studies have shown that if a site's response time is more than 10 seconds, then end users tend to leave. This is generally a bad thing and should be avoided, as there is no secret that a web site's downtime can result in a significant amount of loss to the business.
The second reason is that capacity planning helps you to decide how to allocate resources for a system in terms of the CPUs, RAM, Internet connection bandwidth, and LAN infrastructure needed to support the required performance levels and plan for future growth. Once we understand the limitations of the existing hardware configuration, we can estimate the amount of additional hardware needed to support any increased demands in performance.
Finally, capacity planning is important because it helps to answer the question of what hardware and software infrastructure is required to enable the current system to achieve expected performance objectives.
Chapter 11: Performance Test Reporting
Software Performance Testing Handbook
A Comprehensive Guide for Beginners
A Comprehensive Guide for Beginners
The Performance Test Reporting approach is totally different from the functional test reporting way. In the Performance Testing world, it is not easy to provide the pass/fail criteria as in the case of functional test report. The performance test report development is not as easy as everyone thinks because the test report should contain the symptoms and observations of the performance bottlenecks along with the recommendations. More than that, the information should be abstracted based on the audience of the test report.
Why is it Critical
The Performance Test Report is the document or presentation which contains the details of the performance test outcome. In case of business critical systems, the Performance Test Report would have a direct impact on the business decisions regarding the production release of the application. In some cases, the decisions related to infrastructure upgrades depend upon the test recommendations provided in the Performance Test Report. In case of products, few sections of the Test Report can also shared with all the clients as the performance benchmark. Though we should accept the fact the test report contents purely depends on the test objective, there are certain basic fundamental things which needs to be followed to develop a good test report.
How to report the results of performance testing
The foremost information to be provided in the Performance Test Report should be the overall objective of the performance testing. It should also provide the overview of the application and the business scenarios identified for testing and the user distribution model used for the testing. Though these details are provided in the test plan document, it is a good practice to include these information in the test report.
The main objective of the Performance Testing is to identify the server capacity (Throughput) to handle user load. The business might also be interested in the response time identification for the critical transactions. The Performance Testing activity provides the insight of SPEED, SCALABILITY and STABILITY of the application. The Performance Test Engineer should have this in mind while writing the test report.
The important information that should be available on the Performance Test Report includes the response time details in seconds, server throughput details in terms of transactions per second or Bytes per second , application errors or other errors encountered during the test and an indication about the hardware resource utilization levels.
Mere copying of few graphs from the performance testing tool doesn’t meet the objective of test report creation. The Performance Test Engineer should explain the test results through plain sentences in business English about what is the inference made from each of the graphs provided in the test report. In case of specific bottlenecks, symptoms needs to be explained as all the readers of the test report may not understand from looking at the graph. Each of the graphs should have its purpose detailed along with the clear details of x-axis and y-axis.
The Performance Test Engineer should ensure that right statistics are used and the report is customized to the intended audience. Also the performance test report is expected to have more visuals rather than providing the test results in text format.
Components of a good Test Report
A Performance test report will have different kind of readers like business people (management level people), technical members (architects, designers) and infra members (server admin and DBA). A good test report should meet the expectations of different kinds of readers and should make them understand the test results by adopting right level of abstraction.
It is always a best practice to have a separate section in the test report for providing the test results summary with a high level abstraction of information where technical details needs to be presented at high level but enough information on the test conclusions should be available as major business decisions are made based on this summary.
At a high-level, a performance test report should have the following information readily available.
• Performance Test objectives / goals (Service Level Agreements)
• Test environment and the server configurations
• Tests conducted (along with the deviations from the plan)
• Performance Test Result summary
• System Performance details with enough visuals
• System Performance Bottlenecks details with enough visuals
• Performance test conclusion and recommendations
Why is it Critical
The Performance Test Report is the document or presentation which contains the details of the performance test outcome. In case of business critical systems, the Performance Test Report would have a direct impact on the business decisions regarding the production release of the application. In some cases, the decisions related to infrastructure upgrades depend upon the test recommendations provided in the Performance Test Report. In case of products, few sections of the Test Report can also shared with all the clients as the performance benchmark. Though we should accept the fact the test report contents purely depends on the test objective, there are certain basic fundamental things which needs to be followed to develop a good test report.
How to report the results of performance testing
The foremost information to be provided in the Performance Test Report should be the overall objective of the performance testing. It should also provide the overview of the application and the business scenarios identified for testing and the user distribution model used for the testing. Though these details are provided in the test plan document, it is a good practice to include these information in the test report.
The main objective of the Performance Testing is to identify the server capacity (Throughput) to handle user load. The business might also be interested in the response time identification for the critical transactions. The Performance Testing activity provides the insight of SPEED, SCALABILITY and STABILITY of the application. The Performance Test Engineer should have this in mind while writing the test report.
The important information that should be available on the Performance Test Report includes the response time details in seconds, server throughput details in terms of transactions per second or Bytes per second , application errors or other errors encountered during the test and an indication about the hardware resource utilization levels.
Mere copying of few graphs from the performance testing tool doesn’t meet the objective of test report creation. The Performance Test Engineer should explain the test results through plain sentences in business English about what is the inference made from each of the graphs provided in the test report. In case of specific bottlenecks, symptoms needs to be explained as all the readers of the test report may not understand from looking at the graph. Each of the graphs should have its purpose detailed along with the clear details of x-axis and y-axis.
The Performance Test Engineer should ensure that right statistics are used and the report is customized to the intended audience. Also the performance test report is expected to have more visuals rather than providing the test results in text format.
Components of a good Test Report
A Performance test report will have different kind of readers like business people (management level people), technical members (architects, designers) and infra members (server admin and DBA). A good test report should meet the expectations of different kinds of readers and should make them understand the test results by adopting right level of abstraction.
It is always a best practice to have a separate section in the test report for providing the test results summary with a high level abstraction of information where technical details needs to be presented at high level but enough information on the test conclusions should be available as major business decisions are made based on this summary.
At a high-level, a performance test report should have the following information readily available.
• Performance Test objectives / goals (Service Level Agreements)
• Test environment and the server configurations
• Tests conducted (along with the deviations from the plan)
• Performance Test Result summary
• System Performance details with enough visuals
• System Performance Bottlenecks details with enough visuals
• Performance test conclusion and recommendations
Subscribe to:
Posts (Atom)