Does Maths play a vital role in Performance Testing

Most of you must be puzzled to know Maths (Statistics) plays a vital role in Performance Testing. Yes, we must accept the fact that performance tester should use mathematical knowledge to conduct effective performance testing activity.


Slowly , (i mean it took a long time for us to understand and appreciate that mathematical knowledge is required for Performance testing) we started appreciating the fact that - a new strategical move is there in Performance testing which is nothing but the use of more & more analytical approaches to model the system more realistically and to predict the system performance for varying higher loads.


I would like to share this paper - A strategical move towards Analytical modelling with you. This paper will help you to appreciate the importance and relevance of analytical approaches for effective performance engineering activities. I prepared this paper for STEPIN 2008.

Look for the entire article here...

Capacity Planning - An ITIL Way

I prepared a paper for QAI - ITSM conference. Though it was not selected for the conference, just thought of sharing that paper with you.

This paper will provide you the glimpse of what is ITIL , what is ITSM , What is Capacity Planning, who to assure 100% system availability through effective capacity planning and how to do effective Capacity Planning.

Look for the entire article here...

Hope it helps!!!! Happy learning

A Paradigm Shift in Performance Testing

Scott Barber shares his experience in WOPR6 (sixth meeting of the Workshop on Performance and Reliability), where he happened to meet couple of great people who are known for their work in the Performance Testing field in his August 2007 column. (http://www.perftestplus.com/resources/023PeakPerf.pdf)

He happened to meet lot of great experts including Neill McCarthy, a top performance tester from U.K,Antony Marcano, a performance test consultant in U.K and manager of Testing Reflections.com,Goranka Bjedov senior Software Engineer at Google,Craig Rapin, a performance test manager of world's largest financial services,etc where he got lot of insights about Performance testing and how its gaining maturity day by day.

Am sure that if you read his article on 'Performance Testing Moves toward Maturity' , you will really feel somewhat great to think that you are also in the great research area into which there are already 10-15 pioneers who are doing great things.

Am sure that you will feel motivated.... Now the question is how are you going to withstand with this maturity.. Yes its high time for us to act immediately and to show the maturity in our way of working too :)

Directions for fellow Performance Testers

Before certifying the application performance,
are you satisfied with your own performance? are you sure that you are conducting effective performance test? Have you ever thought how to ensure the your performance test strategy and are you sure that your performance test report is bug free?

The following tips would help you to get more maturity and to do your job better

Yes..You need to be comfortable in the terminologies used in the performance engineering field and need to learn one or two popular and effective performance test tools and develop bottleneck analysis skills. That's the basic skills required for being a performance tester. Then you are matured enough to look back at your performance and look for how effective you are in your activities.

That's where, you need to start looking at Analytical modeling and simulation modeling approaches to move to your next level of expertise.

Foremost thing is that, understand the operational laws and start applying those laws in your day to day test activities.

Secondly start learning about Workload Modeling concepts and Statistical distribution fitting and apply those learning's to define performance testing goals/SLAs.

Thirdly start learning about Demand planning and Performance prediction techniques. Learn Mean value analysis and Erlang's method.

Fourth step is start learning about Capacity Planning.

This is the second level of maturity you need to gain. The next advanced third level is moving towards simulation modeling and advanced capacity planning.

Hope this helps to set your carrer path and plan to work towards it.

What is Capacity Planning in a bird's eye view?

Though the awareness about the Performance Engineering is slowly increasing, still people are thinking that Capacity planning as an alternative to performance testing or sometimes planning for the server capacities before baselining the software performance.

Capacity Planning helps in identifying the specific point of time in future through prediction when the system saturation is going to occur and provides the most cost-effective way of delaying system saturation as long as possible.

Capacity Planners help in identifying the system's saturation point. When and why the saturation would occur and which are the best cost-effective alternatives for avoiding the saturation.

Before looking at the underlying server hardware, we need to first look at the software performance and perform tuning activities to get better performance. Then we need to think about the server capacities.End of the day, its the server's workload pattern which consumes the system resources. Identify the system workload and analyze the system resource usage for the typical workload expected on the server. Based on this, arrive at the capacity baseline on the current server hardware.

Based on the business forecasts, do a prediction on how much increase would be expected in the workload due to the business forecast and identify the required server resources to support the demand. Thus Capacity Forecast is done for the forecasted load expected by the business.

Things to note before you attend interview for Performance Testing role

Based on my experience as a Interviewer, i would like to share few things which the interviewee must follow/know before attending the interviews for Performance tester role.Couple of things which i felt its really missing in many interview candidates is what i have compiled and provided below. I feel that the below skills are mandatory for any performance tester for 1-2 years of experience.

1. Understand that LoadRunner/Silk Performer knowledge is not everything. Its just a tool which enables us to do better performance testing.

2. Be good in Performance testing terminologies - Types of tests and its objectives,various Performance metrics,monitors,etc. The differences/changes required in scheduling each type of tests should be known.Irrespective of whether you have the responsibility for deciding the test strategy , always know when and why to conduct each of the tests.

3. First, be clear on the objective of conducting performance test for any application. I mean set your own objective that you (along with the team) need to make the system to meet its performance objective. Plan realistic tests accordingly as per the business understanding and try your best to identify various load situations that make the system to fail and work as a team to resolve those bottlenecks. Its not enough to conduct some load tests and say that application doesn't meets it Throughput SLA or response time SLA.How you identify and isolate the bottleneck is more important. What and How you observe and interpret during the test is very important.

4. As a Performance tester, you should know the basic relationship between Hits , Response time , Throughput, etc. This forms the basic for the bottleneck analysis. Increase in response time is not a bottleneck you have identified, its the symptom for some bottleneck. Its a must to identify the hidden bottleneck rather than thinking that's its out of scope of your testing. Its your responsibility to run various isolation tests to gather observations about system performance and provide your suggestions to the DB admins and system admins. Dont forget that its your responsibility to get those people involved to help you out in isolating the issues. I agree that, its a team effort but the sole leader here is the Performance tester who need to coordinate all the other members to get the bottleneck identified. Always think that they are going to help you rather than thinking that you are helping them. That thought makes lot of difference in the ownership you have and the responsibilities you hold.

5. If you are an expert in a specific performance test tool, rank your competency appropriately. Explain what you know and what you don't know. State the reasons for what you don't know and why couldn't use\know that. All the organizations will not look for tool expert who knows everything about a specific tool. Having the tool knowledge is important, but that's not everything. So, be frank enough to tell what you know and what you don't know. How much you are conceptually good is what matters in most of the cases.

6. Atleast in Performance Testing/Engineering field, i definitely feel that a good performance engineer should have referred lot of books/articles/white papers if he/she is really looking for a way to conduct more effective performance tests. Talk about how it made a difference in you/your project.

7. On the whole, the performance tester should include the tool knowledge(1 or 2 tools),knowledge and experience for doing performance requirements analysis to set performance SLAs, Test environment creation details , Bottleneck analysis skills, Basic knowledge about the technology used in the applications and basic knowledge about the OS , Server software & hardware details. (I have just stated the core required skills).

Hope it helps you to relook at your skills and identify the gaps before you attend your next interview.

How to conduct effective Performance Tests

Have you thought about statistical distribution fitting of your application's user request arrival pattern ?

Check out the Posters(3 slides) and the paper titled "The Forgotten facts in the workload modeling of an ebusiness application" published in the QAI conference - STC 2007 held at Bangalore on Aug 24-25 2007.

This paper pin points few of the most forgotton facts in the workload modeling of an web application.

Look for the entire article here...

Poster Slides Presented in th QAI Conference - Poster Session
Slide 1 :



Slide 2:
Slide 3 :

Performance Testing -Begineers Reference

For any one who is very new to Performance testing or who is just into performance testing and want to know more......

Scott Barber's Articles/Series about User Experience and Best Practices in performance Testing would help you to set your directions. (http://www.perftestplus.com/pubs.htm)

Infact i started my own way from Scott Barber's above mentioned series. I have read them atleast 3-4 times completely and even now whenever i find time, i still love to read them again as i feel i could get some ideas each time i read them again and again.

Give it a try and track your improvement. If it doesnt works , then write to me. I can help you to make you realize what went wrong.

A Book on Performance Testing for beginners....

Thanks a lot viewers. Thanks for your encouragement by your kind mails.

I am planning to publish a book on Performance Testing comprising all that i learned during the past 3 years of the experience in my career.

I started my carrer as a functional tester and pushed or pulled into Performance testing fortunately thereby i came to know about this area. Mine is a hard way and i didnt had much support in terms of helping hands to get to know about the fundamendals before i started handling the projects. Slowly i learned a lot and finally ended up in unlearning a lot because of the ignorance i had initial days.

The objective of this book is to help fellow performance testers so that they can spend their quality time in achieving great things rather than reinventing the wheel :)


"Performance Testing Handbook - A Comprehensive Guide for Beginners"


The bird's eye view of topics covered in this book includes,

1. Why Performance Testing is crucial.
2. Dynamics and myths in performance testing
3. Deriving Quantitative performance test goals
4. Common mistakes in performance testing which i learned
5. Overview of performance test tools - How to choose the best tool
6. Setting up the performance test environment
7. Comprehensive support for writting test scripts
8. Application benchamarking
9. Quick look at the Industry standard benchmarks
10. A helping in setting up the performance Test Strategy
11. Performance Test Monitoring
12. Performance Test Execution - When and how to choose various tests (Load, Stress, Volume, Soak, Endurance tests).
13.Bottlenecks identification strategy
14.Reporting - How does it matters so much
15.Road Ahead - towards mathematical modeling and queuing theory basics

Web Log Analysis

A typical issue which i faced during web log analysis...

While conducting Performance testing for applications which are falling into the category of Post-Production case, the most important activity that needs to be done is web log analysis. The web server logs are the gold mines which hold all the information about the server during the load condition.

During the web log analysis, look for the peak hour traffic time and identify what is the server load in terms of hits/sec and in terms of users. Based on this statistics, derive performance test goals by considering the business forecasts.

I would like to share a typical issue which I faced while doing web log analysis for a typical post production category of an application. The application under test uses a shared web-app server (IIS) in which there was 2 other applications deployed. Now, the problem is the web server log files have the traces of all 3 applications together as these files are common for a server and not specific for each of the applications running on the server.

The web log analyzer tool which I was using doesn’t have any option to set filters. In that case, I tried using a web log analyzer tool (123LogAnalyzer tool) which provides a filter option to filter out the traces pertaining to a specific application and export the filtered log file into a text file. For example, if the application under test has a context root as ‘methodologies’ (http://132.23.34.21/methodologies/......), then try setting up this URL in the filter and create the text files for your application of interest.

Now you could use the newly generated text file to feed it into any of the log analyzer tool which you are confident enough and proceed with your performance goal setting. My kind advice is that before choosing the log analysis tool of your choice, analyze the accuracy of log analysis thoroughly.

For more details on the various web log analysis tools, look for the web log analysis comparison article available in my blog.

Hope it helps!!!

Proactive Vs Reactive Approach in Performance Testing

Reactive Approach

The Performance Testing activity is often considered as a reactive way of performance management. In most of the cases, the system performance is never thought of during the early phases of Software Development Life Cycle phases (SDLC). Performance is often thought of only as a last activity after the System Testing phase.

And most of the time , the performance bottlenecks revolves around the system architecture or the system design which is a very high cost activity to think about & in certain cases , the system is put into trash because of the huge deviations in the performance constraints.

Basically waiting for the performance problems to appear & then dealing with it at the end is always not a better approach. Mostly Performance Testing is considered as a reactive approach as there is not much importance is given to the system during early life cycle phases. It is more a ‘fix-it-later’ approach which is not that effective.

Proactive Approach

The Proactive approach anticipates the performance problems well in advance & adopts techniques for mitigating them. The Importance of the Performance is thought about from the Initial phase of Software Development Life Cycle (SDLC) & various performance engineering activities are identified for the system.

The disadvantages of ‘fix-it-later’ approach are well understood & engineering practices are adopted to analyze the system design in performance angle.

The integration of performance engineering activities with the SDLC phases is provided in the below table.

Tips for developing effective Performance Test Scripts

The Test scripts are generated to mimic the user behavior during the performance tests. The test scripts enable the performance tester to simulate the real time user behavior by providing the configuration options. The Test scripts needs to be modeled in such a way that it exactly simulates the user actions & user think times.

The Test scripts should have the transaction names or browser requests identified for which the response time & throughput metrics needs to be collected. These requests should be named with proper naming convention so that it becomes easy to refer to a specific transaction by its unique name.

Running the script containing the right set of transactions alone doesn’t mimic the real time user behavior. Each transaction should be always coupled with another important metric called as think time, which is the time taken by the user to think and to navigate between various pages. This wait time which is placed between each transaction determines the accuracy of the scripts simulating the user behavior. Analyzing the think time between each transaction is very critical as it determines the load on the server at any point of time. By conducting discussions or surveys with business analysts & end users of the application, the think time for each transaction can be identified. Say, you have 2 transactions in your script, Login transaction followed by the think time of about 10 seconds followed by the Logout transaction. If the think time is changed to 5 seconds, then the frequency of the login & logout requests send to the server increases, hence the load (requests/sec) on the server increases. If the think time is changed to 20 seconds, then the server load (requests/sec) decreases as the requests are sent in 20 seconds interval.

The test data parameterization is another important activity which needs to be taken care in the scripts. Care should be taken so that enough test data is provided in the script so that each test data is not used more than once when the tests are run. For example, the user login credentials needs to be parameterized in the script. If the test is planned to run for 100 users, then the script should have 100 login credentials available in the script. If more than 1 user is allowed to use the same test data, then the response time would be varying as it might be caching the web pages or the session might get reused. It’s always recommended to make sure that the script is supplied with enough test data to be used uniquely for each user. By using parameterization, we can make sure that the correct formatted unique data is sent from the client-side for each user during the course of the test run. From the server side, there might be some dynamic data generated for each user which may be unique for each user. For example the session id of each user would be unique. If the server passes the session id of the user while returning the response for a specific client side request, then this part of the script cannot replay correctly if it is run for a different user id from the parameterized test data.

This kind of server-side dynamic data needs to be handled in the script as well. Sometimes the recorded script cannot replay because of some specific dynamic values which are generated by the server. This kind of dynamic values needs to be first studied whether it’s unique with respect to each logged in user or unique for each run even for the same user. Sometimes the user action gets completed even if the dynamic values are not handled. Hence, first analyze the logic under which there is a dynamic value getting generated from the server end & accordingly handle the dynamic value. Most of the time the dynamic value is handled by identifying the corresponding left & right boundary & capturing the dynamic value, then the captured value can be substituted against the variable which is holding the dynamic value.

Last but not least, make sure the script is very realistic in terms of the user actions. Try to have a single script which covers all the flows of the application & use random number generator function to distribute the user load across various functional flows. Sometimes it’s good to have multiple scripts as long as the distribution of the load across scripts is handled separately.

Happy Scripting!!

How to run effective performance tests

To plan for an effective performance test, follow the below four important steps

1.The first step is to understand the web site’s actual usage pattern. Mostly this information can be collected from the web server log files which are considered as the ‘gold mines’ to know every possible detail about the server load. The ‘log analyzers’, tools which helps to analyze the web log files and provide the analysis report in terms of number of user sessions, number of unique users, number of errors, peak day traffic, peak hour traffic, user request arrival pattern, frequently accessed pages, user navigation pattern, etc. From this analysis, the actual usage of the server can be identified.

2.The second step is to design the performance test approach. Based on the business forecast of the user load increase, develop service level agreement (SLAs) that has to be met by the application in order to handle the user load in the target environment. The analysis done in the first step helps to set quantitative SLA for the application. Design the test cases that needs to be tested along with the parameters like ramp up strategy, test duration, think time settings, test scenarios to be tested, different load conditions & types of tests along with the pass/fail criteria. Baseline the test approach & use this as the reference for the rest of the activities.

3.The third step is to identify the performance test tool to help us in simulating the user load on the server. Look for performance test tools available in the market (Licensed or freeware tools) based on the budget constraints & technologies supported by the test tool. Using the test tool of choice, develop the test script which simulates the actual user access pattern. Have enough test data available in the test script in order to simulate different users using different data available in the script.

4.The fourth step is to look for the readiness of the test environment. The test environment includes the application’s server infrastructure & also the load test environment. Enough care should be taken to ensure that no other users access the application during the test run & the servers are isolated from any other usage. In the case of load test environment, ensure that load generators are in place to generate the required load, the availability of test tool license to support the load to be simulated, test scripts are developed and ready with the required test data & think time, server monitors are configured with the required performance counters & the server admins are informed for their assistance in case of server failures.

Choose right set of test scenarios to schedule the performance test

Like functional testing, performance testing cannot be done for all scenarios. Identify the frequently used scenarios based on discussions with the business analyst. Sometimes, there will be couple of scenarios which have some suspections about its performance. We have to include those kind of scenarios too. It boils down to Pareto analysis – the system performance will be improved if 20% of the scenarios which are most frequently accessed are tuned. Tuning the remaining 80% of scenarios will not bring considerable performance improvement. Use Scott barber’s UCML or CBMG modeling to identify the critical scenarios.

During the performance test, don’t forget that you need to create the actual real time system usage model. (i.e.) The test should be very realistic in such a way that it models the real time traffic & more important is it should model the transaction mix of the real time workload. If you are scheduling a test for 100 users, then make sure that the 100 users are distributed across various functional flows of the application (Say in a banking application during the peak hours the users are distributed across various functionalities like, 60% of the users depositing the amount, 20% of the users transferring the amount, 10% of the users checking for the account balance & 10% of the users accessing the mini-statement). Unless this kind of mixed mode scenario is adopted for the testing, we might not end up simulating the real time user load.

There are certain situations wherein we might need to schedule the performance test with a single functional flow. One situation is when the performance testing is scheduled for each of the functionally stable build. Other situation would be when we identify some performance issues during the mixed mode scenario, then in order to isolate whether the performance issues are from a specific functional flow, we might need to run single functional flow test.

Observe the differences quoted above. As long as your objective is to test the system performance for a specific load, make sure the load is very realistic to the real time user access.

Metrics to be collected during any Performance Tests

Irrespective of the kind of performance test (load test/stress test/volume test/spike test etc), there are certain metrics which needs to be collected from the performance test tool to analyze the system performance.

End of the day , we want to know whether the system is able to handle the load meeting the response time level SLA. To achieve that we should identify the following 2 things.

1. Whether my software pipe is able to effectively utilize the available hardware pipe.

2. Whether my hardware pipe is big enough to be used by the software pipe.

If the software pipe is not able to utilize the hardware pipe effectively, then no wonder that even if you upgrade the server hardware, there will no improvement in the system performance.

Metrics to be collected

1.As a first step always look for the average & 90th percentile response time of all the transactions. Just neglect if there is other transactions which are inserted by the tool itself. (In case of Load Runner, the tool provides the transaction response time for vuser_init, vuser_end & action transactions depending upon the run time settings).
2.Next look for the load generated from the test tool. Hits/sec is the metric which provides the information on how many requests are fired to the server per second. Look for the average hits/sec & if the test is run using a specific ramp up rate, then measure the hits/sec value during the stress period of the test(when all users where running).
3.Next look for the Throughput, the metric which portrays the load handled by the server. The total transactions passed per second metric helps to identify the how capable is the server to handle the transactions. Also look for throughput in terms of bytes/sec metric which again portrays the server capability to respond to the client requests.
4.Next look for the failures occurred during the test. Observe when the failures have occurred, what is the type of failures (network failure, HTTP 500 errors, HTTP 404 errors, proxy server failures, etc) & is the failures specific for a transaction.
5.Next, look for the basic performance counters of web/application/DB server resources like %CPU, processor queue length, Memory pages/sec, Page faults/sec, % Disk utilization.

These five are the basic performance metrics which are mandatory for any performance tester to analyze the performance of the system. Going forward, in order to isolate the performance bottlenecks, we might end up looking at various performance metrics depending upon type of the performance issue.

Know the importance of Think time

In most of the performance testing approach, the ‘Think Time’ metric is taken for granted. We should understand the importance of think time before deciding the test strategy.

What is a Think Time? Think time is the time taken by the users to think or to navigate
to different pages in the application.

Depending upon the application context, the think time would vary. Its not advisable to have the default think time for all applications under testing. For the user registration page containing 5 mandatory fields, the use might take 15-20 seconds on an average. Notice the words “15-20 seconds on an average”. You can’t say that all the users would take exactly a specific think time. It would definitely vary based on each of the users. So, always make a point that whenever you run the load test, apply a randomization factor on the configured think time. Say if the think time in the script is 10 seconds between each of the transactions, configure the load test tool to use a randomized think time of 80% to 120% of the value available in the script.

As we are interested to measure the response time of the transaction (user defined actions), care should be taken to place the think time outside the transaction points. (i.e.) the think time should not be placed between the start & end of the transaction.

Most of the testers doing the performance testing don’t understand that by changing the think time, we can simulate the load of large number of users though we run the test for less number of users. To illustrate, consider the following example. Five tests were conducted with 5 virtual users simultaneously accessing the application. In each of the test, the think time is the only metric which was changed. Observer, the Hits/sec generated from the client side & Throughput (the number of transactions passed /sec).

Though the number of users hitting the server is same, the think time creates a huge impact on the server load. We need to understand the fact that decreasing the think time will increase the server load & vice versa. Always try to model the real time scenario & identify the performance figures. Never forget that we can play around with the server load just by changing the think time.

Also, dont forget to use Little's law, N = X . (R+Z) to check whats the actual load on the system for the test which you are running.

Relationship between Metrics

Things to note : Queuing Theory basics

1. The utilization of a service center is dependent on the Arrival rate.
2. The Service Demand remains constant irrespective of the load (for Load Independent devices).
3. Throughput increases with the Utilization.
4. Queues will be formed even when the utilization is less than 100% & its also possible that there can be no queues during 100% utilization too. Hence there is no relationship between the Utilization & Queue length.
5.The response time does not depend on the Utilization. It depends on the Queue length, hence the arrival pattern.
6. The maximum CPU utilization that can be achieved can be calculated using the formula Queue length / Queue Length + 1.
(i.e) For a four CPU m/c maximum utilization that can be achieved = 4/5 = 80%

Importance of Little's Law

Little's law is quite simple and intuitively appealing.

The law states that the average number of customers in a system (over some time interval), N, is equal to their average arrival rate, X, multiplied by their average time in the system, R.

N = X . R (or) for easy remembrance use L = A . W

This law is very important to check whether the load testing tool is not a bottleneck.
For Example, in a shop , if there are always 2 customers available at the counter queue , wherein the customers are entering the shop at the rate of 4 per second , then time taken for the customers to leave from the shop can be calculated as

N = X. R

R = 2/4 = 0.5 seconds


A Simple example of how to use this law to know how many virtual users licenses are required:

Web system that has peak user load per hour = 2000 users
Expected Response time per transaction = 4 seconds
The peak page hits/sec = 80 hits/sec

For carrying out Performance tests for the above web system, we need to calculate how many number of Virtual user licenses we need to purchase.

N = X . R
N = 80 . 4 = 320

Therefore 320 virtual user licenses are enough to carry out the Load Test.

Kendall's Notation

In Queueing Theory , Kendall's Notation is used to describe or classify the queuing models.

A / B / C / ( D / E / F )

A - It represents the arrival pattern of the requests to the system.

B - It represents the servicing pattern the system.

C - It represents the total number of servers in the system.

D -It represents the total number of requests that can be queued in the system.

E -It represents the calling source (i.e) the customer base

F -It represents the service discipline of how the job that gets served by the system.

Example :

1. M/M/1/20/30/FIFO = markovian arrival rate , markovian service time , 1 server queuing model , maximum queue length as 20 , user base as 30 , service discipline as First In First Out (FIFO).

Fundamental laws of Performance Testing

Sometime back , i could realize how I was struggling to isolate simple Performance bottlenecks.

Though i managed to identify performance bottlenecks,I know that i was missing something..but really not sure what was it. Pea (Performance Engineering Associates) helped me to identify those missing pieces. Thanks to Pea.

Yes..Its the basics about Queuing Theory.

PEA was formed by professionals who have had the opportunity to lead assignments that brought about several radical changes by bridging the gap between theory and practice in PE.

Fundamental Laws
~~~~~~~~~~~~~~

1. Utilization law
2. Little's law
3. Response Time law
4. Forced Flow law
5. Service Demand law

Check out the free online book on Queueing Theory available in pea's web site.

http://www.pea-online.com/resources.htm

Nice article from the book "Performance by Design - Computer Capacity Planning by Example" by Virgilio A.F. Almeida, Lawernce W.Dowdy, Daniel A.Menasce.

http://www.informit.com/articles/article.asp?p=170721&seqNum=2&rl=1

Watch out for more information about Performance Engineering Vedas shortly.

Other Useful Links about Performance Testing/Engineering

Sites with more valuable information about Performance Testing / Engineering :

http://dev2dev.bea.com/pub/a/2005/09/performance_testing.html

http://www.stickyminds.com/sitewide.asp?Function=edetail&ObjectType=ART&ObjectId=6221

http://channel9.msdn.com/wiki/default.aspx/PerformanceWiki.PerTestingHowToModelUserExperience

http://cs.gmu.edu/faculty/menasce.html

http://www.informit.com/guides/content.asp?g=java&seqNum=267&rl=1

http://www.teamapproach.ca/trouble/MemoryCounters.htm

http://www.tkn.tu-berlin.de/~awillig/user_includes/pet_ss2005/pet_ss2005.html

http://www.perfeng.com/

http://www.wilsonmar.com/perftune.htm

http://loadtest.com.au/types_of_tests.htm

http://www.new-destiny.co.uk/andrew/past_work/queueing_theory/

http://www.pea.co.in/resources.htm

Learn about Performance Testing from Scott barber's Site

Who said Performance testing is just simply simulating more number of users to test the system performance.......

There are lots more than that......Words cant explain the worth of information available in Scott Barber's site.

http://www.perftestplus.com/pubs.htm

Starting from how to plan for a Performance Test till how to create the Performance Test Report are available in a very easy & understandable way. Its amazing.

Thanks to Scott barber for sharing the information with us.

Performance Test Estimation Approach

Challenge your Performance Test Estimation Effort Using Test Factors

Performance Test effort estimation is the initial challenge we face in the Performance Test Life cycle. As Performance Testing is more like an investigation activity which may take any direction, the ad-hoc estimation methodologies used for performance test estimation doesn’t seem fruitful. Because of the dynamism and varying scope, there are lots of issues in the test effort estimation. Based on our experience in the performance testing projects on the web based applications, we conjecture that the Performance Test Estimation approach using Test Factors could be more reliable. Our approach is based on the consideration of the various test factors involved in scripting, execution and analysis, thereby eliminating the use of ad-hoc estimation methodologies and it is architecture / technology independent.

This paper describes how to do effort estimation for a performance testing project. The paper focuses on the pre-requisites and the factors which need to be analyzed in order to arrive at the performance estimation model along with the details about how to use this estimation model. This approach is being implemented in Honeywell for the estimation of Performance Testing effort on web applications and is proven to be ninety-five percent accurate. It’s been used to estimate ten projects falling under simple, medium and high complex project categories. The case study illustrates the application of this technique to arrive at the estimation for a typical web application.

The readability of the paper covers Performance Test Engineers, Test Leads, Test Managers and others with the domain knowledge of performance testing.

Look for the entire article here...

Network Bottleneck Symptoms

The simplest way to measure effective bandwidth is to determine the rate at which your server sends and receives data. Network bandwidth availability is a function of the organization's network infrastructure. Network capacity is a function of the network cards and interfaces configured on the servers.

Network Interface: Bytes Total/sec : To determine if your network connection is creating a bottleneck, compare the Network Interface: Bytes Total/sec counter to the total bandwidth of your network adapter card. To allow headroom for spikes in traffic, you should usually be using no more than 50 percent of capacity. If this number is very close to the capacity of the connection, and processor and memory use are moderate, then the connection may well be a problem.

Web Service: Maximum Connections and Web Service: Total Connection Attempts : If you are running other services on the computer that also use the network connection, you should monitor the Web Service: Maximum Connections and Web Service: Total Connection Attempts counters to see if your Web server can use as much of the connection as it needs. Remember to compare these numbers to memory and processor usage figures so that you can be sure that the connection is the problem, not one of the other components.

To determine the throughput and current activity on a server's network cards, you can check the following counters:

· Network\Bytes Received/sec
· Network\Bytes Sent/sec
· Network\Bytes Total/sec
· Network Current Bandwidth

If the total bytes per second value is more than 50 percent of the total capacity under average load conditions, your server might have problems under peak load conditions. You might want to ensure that operations that take a lot of network bandwidth, such as network backups, are performed on a separate interface card. Keep in mind that you should compare these values in conjunction with Physical Disk\% Disk Time and Processor\% Processor Time. If the disk time and processor time values are low but the network values are very high, there might be a capacity problem. Solve the problem by optimizing the network card settings or by adding an additional network card. Remember, planning is everything—it isn't always as simply as inserting a card and plugging it into the network.

Disk Bottleneck Symptoms

A bottleneck from a disk can significantly impact response time for applications running on your system.
Physical Disk (instance)\Disk Transfers/sec counter for each physical disk and if it goes above 25 disk I/Os per second then you've got poor response time for your disk.
By tracking Physical Disk(instance)\% Idle Time, which measures the percent time that your hard disk is idle during the measurement interval, and if you see this counter fall below 20% then you've likely got read/write requests queuing up for your disk which is unable to service these requests in a timely fashion. In this case it's time to upgrade your hardware to use faster disks or scale out your application to better handle the load.

Look for the Physical Disk (instance)\Average Disk Queue length & Physical Disk (instance)\Current Disk Queue length parameters to get more details on the queued up requests.

CPU Bottleneck Symptoms

Symptoms for CPU bottlenecks include the following :

The Processor(_Total)\% Processor Time(measures the total utilization of your processor by all running processes) will be high. If the server typically runs at around 70% or 80% processor utilization then this is normally a good sign and means your machine is handling its load effectively and not under utilized. Average processor utilization of around 20% or 30% on the other hand suggests that your machine is under utilized and may be a good candidate for server consolidation using Virtual Server or VMWare.

Further to breakdown this %processor Time , monitor the counters - Processor(_Total)\% Privileged Time and Processor(_Total)\% User Time, which respectively show processor utilization for kernel- and user-mode processes on your machine. If kernel mode utilization is high, your machine is likely underpowered as it's too busy handling basic OS housekeeping functions to be able to effectively run other applications. And if user mode utilization is high, it may be you have your server running too many specific roles and you should either beef hardware up by adding another processor or migrate an application or role to another box.

The System\Processor Queue Length(indication of how many threads are waiting for execution) consistently greater than 2 or more for a single processor CPU is a clear indication of processor bottleneck . Also look at other counters like ASP\Requests Queued or ASP.NET\Requests Queued as well.

Code & Application server related performance issues in J2EE

In J2EE environment, there are some common Code related or Application server related problems. It include :

Code related problems :
~~~~~~~~~~~~~~~~~~
1. Slow Methods
a. Consistently Slow Methods
b. Intermittently Slow Methods
2. Synchronization Problems
3. Memory Problems
4. Coding Practices, such as using exceptions as a means to transfer control in the applications

Application server configuration problems :
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1. JDBC Connection Pool size
2. JVM Heap size
3. Thread Pool size

Symptoms for Application & Web Server Bottlenecks

Symptoms for Application server bottleneck
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1. Increased 'Server Time' breakup
2. One or more page components of transaction takes more time where in the DB query is having less execution time.
3. The Static files are having less response time whereas the dynamic contents (servlets,jsp,etc) takes more time.
4. Network delay is negligible.
5. Home Page gets displayed in few seconds even during the stress period(as it is fetched from the web server).
6. Hits / sec & Throughput remains less.
7. If the CPU/ Memory/Disk of the App server has any bottleneck symptoms.
8. If the HTTP / HTTPS connections established doesn’t increase proportionally with the load.
9. If the new connections established is very higher & the reused connections are very less.

Symptoms for Web server bottleneck
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1. Increased 'Server Time' breakup
2. One or more page components of transaction takes more time where in the DB query is having less execution time.
3. The static files are having high response time than the dynamic contents (servlets,jsp,etc).
4. Network delay is negligible.
5. Home Page takes more time for display.
6. Hits /sec in the web server is very less.
7. If the CPU/ Memory/Disk of the web server has any bottleneck symptoms.

A Comprehensive Report on Web Log Analysis

In today’s dynamic world, the IT sector of all the businesses want to use more sophisticated techniques to support their application scalability. All the web site owners are interested to know the usage pattern of their site in order to make any business decisions. The necessity to know the visitor profile is very high in any web site as most of time the visitors are anonymous & unpredictable for any open systems (which uses internet).

The Web analytics activity deals with measuring the end user behaviors on a web site. Of the two major web analytics techniques (Web server log analysis & Page Tagging) available, this paper deals with the Web Server Log Analysis concepts. This paper provides an overview of various log analyzer tools available in the market & their comparative ratings.

The Web log analysis is used by various types of users with different backgrounds. The different sets of users are interested to know about specific data from the web server log. This has lead to the development of lot of log analyzer tools with specific extra features, though the basic features are available in most of the tools.

Look for the entire article here...

Hardware Malfunctioning Symptoms

System\Context Switches/sec (measures how frequently the processor has to switch from user- to kernel-mode to handle a request from a thread running in user mode.). If this counter suddenly starts increasing however, it may be an indicating of a malfunctioning device, especially if you are seeing a similar jump in the Processor(_Total)\Interrupts/sec counter on your machine.
You may also want to check Processor(_Total)\% Privileged Time Counter and see if this counter shows a similar unexplained increase, as this may indicate problems with a device driver that is causing an additional hit on kernel mode processor utilization.
If Processor(_Total)\Interrupts/sec does not correlate well with System\Context Switches/sec however, your sudden jump in context switches may instead mean that your application is hitting its scalability limit on your particular machine and you may need to scale out your application (for example by clustering) or possibly redesign how it handles user mode requests. In any case, it's a good idea to monitor System\Context Switches/sec over a period of time to establish a baseline for this counter, and once you've done this then create a perfmon alert that will trigger when this counter deviates significantly from its observed mean value.

Memory Bottleneck Symptoms

When it comes to the System memory , there are 3 things to monitor :

1.Monitor Cache (Hits/Misses),
2.Monitor Memory (Memory Available/sec, Process/Working Set),
3.Monitor Paging (Pages Read/Sec, Pages Input/Sec, Page Faults/Sec, % Disk Processing)

Memory\Available Bytes, if this counter is greater than 10% of the actual RAM in your machine then you probably have more than enough RAM and don't need to worry.

The Memory\Pages/sec counter indicates the number of paging operations to disk during the measuring interval, and this is the primary counter to watch for indication of possible insufficient RAM to meet your server's needs.

You can monitor Process(instance)\Working Set for each process instance to determine which process is consuming larger and larger amounts of RAM. Process(instance)\Working Set measures the size of the working set for each process, which indicates the number of allocated pages the process can address without generating a page fault. A related counter is Memory\Cache Bytes, which measures the working set for the system i.e. the number of allocated pages kernel threads can address without generating a page fault.
Finally, another corroborating indicator of insufficient RAM is Memory\Transition Faults/sec, which measures how often recently trimmed page on the standby list are re-referenced. If this counter slowly starts to rise over time then it could also indicating you're reaching a point where you no longer have enough RAM for your server to function well.

A Paper on Risk based Testing

Dare to Gamble - Creating a new vibe in Risk Based Testing

Abstract

Uncertainty introduced by the testing method is virtually unavoidable. Testing activities can fail in many ways; however, you can prevent most problems through Risk analysis in the early test life cycle. Risk implies a potential negative impact on any of the test activities affecting the system either directly or indirectly. Steve Wakeland defines IT risk as ‘the likelihood that a program fault will result in an impact on the businesses’.

Risk based testing differs from the traditional way of system testing because it emphases on the various risks involved in the testing life cycle. Though the probability of the risk is known, the outcome remains unknown and hence we could define Risk as the product of the defects and probability.

The objective of our paper is to put forth the risk based test strategy which would help testers to identify the risks involved at the earlier stages of a test life cycle and the reduction of COPQ. The paper deals with the risk prediction and mitigation methodologies. The Metrics Based Management approach illustrated in our case study enabled us to identify the probability and the consequences of individual risks during the test life cycle.

A risk analysis was performed and the functions with the highest risk exposure, in terms of probability and cost, were identified. Risk prediction is derived from planning, assessing and mitigating risks. Risk prediction involves forecasting risks using the history and knowledge of previously identified risks. The risk mitigation activity is done to inspect and focus testing on the critical functions to minimise the impact a failure in the function.

The readability of the paper covers test engineers, test leads and test managers and others with the knowledge of software testing.

1. Introduction

Testing is the time consuming part of software engineering & it is the last phase in the Software Development Life Cycle .It has to be always done under severe pressure. It would be unthinkable to quit the job, to delay delivery or to test badly. The real answer is a differentiated test approach in order to do the best possible job with limited resources.

For getting more confidence that the right things are tested at the right time, risk based testing can help, as it focuses and justifies test effort in terms of the mission of testing itself. The basic principle of Risk Based testing is to test heavily those parts/components of the system that pose the highest risk to the project to ensure that the faultiest areas are identified. At the system level, one probably has to test first what is most important in the application & secondly, one has to test where one may find most defects.

This paper provides a method for Risk Based Testing. The test approach is explained in the following sections. The section 2 details on the basics of Risk based testing; the section 3 provides the high-level test process proposed for Risk based Testing; section 4 illustrated our test strategy using a case study.

2. Risk Based Testing
2.1 Risk
Risk is a possible failure, unwanted event that has negative consequences. According to
Wikipedia , “Risk is the potential future harm that may arise from some present
action.” While talking about risk, we need to think about the cost of it & the probability that it might occur.

Risk = Cost * Probability
Normally Risks can be accepted, prevented or transferred. We need to look at the degree to which we could change the outcome. Risks are divided into three general types: project, business, and technical risks.

2.2 What is Risk Based Testing
The important question that everyone has is “What makes Risk Based Testing different from traditional testing”. We all know that money is important in life, but the importance of it is different from people to people. There are lot of differences between the people who look for job satisfaction in terms of getting a challenging work to earn money & there are people who work for organizations only because of the high compensation package. Though Risks is identified in traditional way of testing, more emphasis on the Risk & building a test strategy around the identified risk makes Risk based Testing different from the Traditional Testing.

Most risk based testing is a black box testing approach .This technique checks the system against the requirement specification. Once the high prioritized risks are identified, testing strategy needs to be developed to explore on those high priority risks.

Our approach is more based on the Risk Analysis activity model shown in Figure 1. This model is taken from Karolak’s book “Software Engineering Risk Management”.

3. Our proposed method for Risk based Testing
Our approach concentrates on the project level technical risks The Business level risks & Management level risks identification/tracking/mitigation is not a part of our approach. This section provides an overview of the various steps involved in the Risk based Testing methodology. The Case study provided in the section proves as the Proof of concept for the below described model.

The Test strategy includes the following steps:

3.1 Risk Identification

3.2 Risk based Testing strategy

3.3 Risk Reporting

3.4 Risk Prediction

3.1 Risk Identification

A risk is an unwanted event that has negative consequences. It’s highly important that the risks analysis should start from the starting of the testing life cycle.

3.1.1 Requirements Risk
Normally, the software testers review the requirements documents before starting the test case designing. Most of the time the requirements stay absurd or contradictory or misleading. Lot of research is going on how to measure the quality of requirements document. Here is a simple way of reducing the requirements level risk which leads to high COPQ when uncovered during later phases of testing. The assumption of this model is that the requirements document has more than 95% coverage of the product requirements & the tester’s role is to verify the requirement document for testability.

1.The requirements should not have any TBDs or TBAs.
2.Each requirement should be rated for the understandability, clarity & testability in the scale of 1 to 5 where 5 being the idealistic value. The product of the U (understandability), C (clarity) & T (testability) factors gives the risk factor for the requirement. The minimum threshold value for the risk factor is 45. Any requirement which is having the risk factor less than the minimum threshold value needs review & revision.
3.Each Module / Component should be rated for the requirements coverage on the following areas – Functionality requirements (FR), Usability & User Interface requirements (UR), and Security & Performance requirements (S&PR). 4.The product of FR, UR & S&PR gives the risk factor for the module or component. The minimum threshold value for the risk factor is 45. Any requirement which is having the risk factor less than the minimum threshold value needs review & revision.
5.Schedule a SGR (Stage Gate Review) with the development team, testing team & the other stakeholders of the project to review & revise the individual requirements and the modules which has not met the minimum threshold value.

Most of the requirements based risks are eliminated by following the above 4 steps. All of us should accept the fact that the risks in the Requirements are the most severe.

3.1.2 Customer CTQs based Risk Matrix
The various risks (project, business, and technical risks) need to be analyzed before starting the test planning. The following are the set of activities which needs to be followed in this phase. Software Testers needs to validate & confirm the Customer’s CTQs against their system understanding.
1. The tester needs to develop a QFD (Quality Function Deployment - Six sigma tool) to prioritize the relevant Testing Types(Functional Testing, User interface Testing , Security Testing, Performance Testing, etc) required for testing the project to meet the customer CTQs(Critical To Quality).
2. The tester needs to develop another QFD to prioritize those modules/components which needs immediate testing in order to meet the customer CTQs.
With the prioritized list of Modules/Components based on customer CTQs , Customer Priority based Component Risk Matrix needs to be developed. It is similar to ‘Component Risk Matrix’ as per James Bach style, but the difference being more emphasis on the Customer expectation based Risk prioritization.

SWOT Analysis technique & Assumption risk mitigation activities help in uncovering lot of technical risks. Clear communication of the risk is as important as the Identification of risks. The Customer Prioritization based Component Risk matrix will provide the clear details on the set of risks in each component of the system.

3.2 Risk based Testing Strategy
The High risk components identified from the above activity are sorted based on the Risk Factor. This list would serve as the input for carrying out the activities in this phase.

For each Component, identify the Smoke test cases which would uncover more defects in the risk prone areas identified in ‘Customer Priority based Component Risk Matrix’. Hence the risk based smoke test case is different from the normal smoke test cases which are developed in Traditional testing.
Each Test case should carry a RPN value (Risk Priority Value). For calculating the RPN value of a test case, provide the rating against Severity of the risk , Probability of risk occurrence & Detection capability of the risk (the ease in detecting the risk). The Range is defined as 1, 5, 9 where 9 being the highest.
The Test cases with RPN values listed against the release versions can be shared with the project stakeholders.
The Management & Customers could decide on the whether specific module level test cases needs to be prioritized or the Smoke test cases with the highest RPN values.

3.3 Risk Based Reporting
Risk reporting is based on the smoke test cases (risk based) identified during the previous topic. The defects uncovered during smoke testing needs to be reported in a meaningful way. It’s important to closely monitor the errors & collect the following metrics:

· Number of Smoke Test cases planned vs. executed vs. failed.
· The test effort planned vs. actual.
· Number of Planned releases(to testing team) vs. number of actual releases
· Number of rejected builds
· Number of resources working on the project
· Total number of defects identified during smoke testing.
· Total number of defects rejected.
· Number of rejected defects due to functional changes
· Number of rejected defects due to lack of understanding
· Total number of defects per module.
· Percentage of tests passed
· Percentage of tests failed
· The effort(in hours) required to identify a defect
· The average time interval between each defect.
· Total number of defects that does not has a test case
· Total number of test cases added at run time
· Number of defects per developer(if possible)
· Number of defects per defect type
· High Priority defects found vs. Medium or Low priority defects
· Total number of deferred defects
· Metrics about Defect Injection phase & defect reason would add value

The identified defects needs to be tracked against the risk of not fixing & it would help the development team to fix the high risk defects. The defects can be plotted as shown in the below graph (Figure 2) for easy interpretation of high priority risky defects. The defects listed in the top right corner marked as “1” needs to be concentrated first.
3.4 Risk Prediction

Risk prediction is derived from the previous activities of identifying, assessing & prioritizing, mitigating and reporting risks. Risk prediction involves forecasting risks using the history and knowledge of previously identified risks.

During test execution it is important to monitor the quality of each individual function & the metrics collected during the testing phase becomes the data for forecasting future risks.

4. The Case Study
The rest of this paper will discuss a case study using the risk based approach to software testing, relating the different activities discussed in the section 3.

4.1 System under Test
The Early Indicators Product Tracking is a process to track the performance of the in-service products based on Reliability analysis. This helps in proactively identifying the in-service products that are experiencing degraded reliability performance. This results in efficient spares management and warranty issues.

The system interacts with various types of databases (SQL, Oracle, COBOL & FoxPro) & collects the various details about the aircraft parts & derives the reliability of various parts.
The system accepts the data from various vendors & does a reliability analysis & provides various charts & reports which enable the vendor in identifying reliability factor of the part.

4.2 Challenges in Testing
The following are the key challenges in the project.

1. Limited Testing Time: As the project required new complex algorithms & services to be implemented, the development timeframe was more than the expected & hence the testing phase needs to squeeze. So it’s very critical to adopt a test strategy wherein the more critical core module which represents more risk needs to be tested first.

2. Limited Testing Resource: There were not much senior testing resources available at the time of the project. The project demands more competent test engineers who have knowledge on the System architecture as the testing involves testing of simulation model & window services. It is highly essential to plan for a resource to complete the testing as soon as possible without putting in efforts for Knowledge transitions or trainings.

3. Simulation Testing: It needs a White box testing approach as most of the time, the tester needs to adopt the reverse engineering strategy to the available data & attain the data integrity.

4. Database Relativity: The system talks with 4 different types of databases (SQL, Oracle, and FoxPro & COBOL) & tracks the reliability of the aircraft parts.

5. Integration with Third party Tools: As the system exposes an interface to interact with the Third party tools like Minitab, thorough testing on the critical areas is required. The limitations of the Third party tools should not interfere in the system testing.

4.3 Risk based Test Strategy
The EIPT application consists of seven modules. The following activities are done as part of the Test Strategy.

4.3.1 Requirements Risk identification
During the requirements document review by the testing team, the requirements of the above mentioned modules which are adhoc & absurd were identified & highlighted. Thus the requirements are validated & revised as per the Requirements Risk mitigation strategy. Each of the modules which lack in functional requirements coverage or Usability & User Interface requirements coverage or Security & Performance requirements coverage were identified & highlighted by following the strategy mentioned in section 3.1. The below table (Figure 3) provides the sample of how the module level requirements are validated & risk factor is derived.

Figure 3 : Module Risk Factor Calculation
After the end of the preparing the list of requirements & module which are vulnerable to failures, Stage Gate Review (SGR) was scheduled to review & revise the requirements. By validating the requirements against the below sample points would uncover a lot of risks:

Is any tool used for creating prototypes
Configuration Management tool not being identified for placing the project artifacts
Software tools to support planning & tracking activities not being used regularly
Requirements put excessive performance constraints on the product
Unavailability of tools or test equipment
Incomplete interface descriptions
Operation in an unfamiliar or unproved hardware environment causes unforeseen problems
Strict requirements for compatibility with existing systems require more testing, and additional precautions in design and implementation than expected

4.3.2 Customer CTQs based Risk Matrix
The objective of this activity to identify the prioritized set of modules based on Customer CTQs & analyzing /identifying the risks available in those modules. Normally the risk prioritization strategies concentrate on prioritizing the risks based on risk severity, meeting the deadlines, failure mode use cases, etc. Our proposed method is found to be more advantageous than the available methods as it concentrates & prioritizes the risks based on the customer expectation & delivers quality.

The VOC (Voice of Customer) was identified by having discussion with the customer & it helped in validating the customer expectations with the QFD developed during requirements phase of the project.
Then in order to prioritize the type of testing required for meeting the customer expectation, QFD was done. The relative rating as shown in the figure 4 shows that Functional testing is the high priority as per customer’s expectation.

Figure 4 : Prioritizing Testing Type using QFD Tool
In order to prioritise the modules for meeting the customer expectations, QFD was done. The relative rating showed the high priority module that needs more concentration as shown in Figure 5.

Figure 5 : Prioritizing the Modules using QFD Tool

The Customer priority based Component Risk Matrix was developed to identify the module level risks & the Risk factor is derived in order to prioritise those risks which are of interest to the customer. For identifying the risks in each module SWOT analysis & Assumption risk analysis are used. The below table(Figure 6) provides the list of risks in each module & the corresponding risk factor.

Figure 6 : Customer priority based Component Risk Matrix

4.3.3 Risk based Test case Design

After the risk assessment & prioritization, smoke test cases were identified against each of the modules. These smoke test cases emphasis on testing the risk areas / fault prone areas of the module. Each test case carry the 3 values - Severity rating(S) , Probability rating(P) & the Detection capability rating(D) & the RPN value was derived for each of the test cases.

The below table lists the sample set of test cases of various modules with their corresponding RPN values.

Figure 7 : Risk based Testcase Design Summary

4.3.4 Risk Reporting

The smoke test cases developed during Test case design phase were executed during testing phase & there were lot of defects logged against each of the modules in a defect tracking tool.

Figure 8 : Risk Report Metrics

4.4 Our Risk based approach - Credits

Our risk based test approach has lot of advantages compared to the traditional test approach.

1. Our risk based approach consumes less resource than the traditional approach. The resource profiles of the Traditional approach, Risk based approach (planned) & Risk based approach (actual) are provided below. As risk based testing emphasis more on identifying the risk areas & uncovering the defects in those error prone areas, the time taken to test those specific core areas are less than the Traditional test effort.

Figure 9 : Resource Profile Comparison

2. The count of high priority (Critical or Catastrophic defects) defects found using the risk based approach is higher than the traditional approach. Though the total number of defects found using the traditional approach might be higher than the risk based approach, more severe defects are identified in Risk based approach as it emphasis on identifying critical defects before interim releases.

Figure 10 : Defects Comparison

3. Another success criteria of our risk based approach depends on having an efficient, effective & flexible test organization.

4.5 Limitations of our Risk based approach

The Risk based approach might not be applicable for newly formed team or inexperienced software testers as it needs more risk foreseeing/assessing competency.
Our approach might not yield great results for projects wherein the cohesion between the system modules are very high & it’s tough to isolate a risk easily.
Our approach might not be the best for projects which does not follows waterfall model of software development.

5 Conclusion

Stephen Besson says that “lowering the risk to 50 percent can be achieved in a shorter period of time if testing prioritization has been done with risk in mind ”. Risk-based testing is a highly effective testing technique that can be used to find and fix the most important problems as quickly as possible. If done well, it allows the test function to expend relatively more effort on the parts of the system that are higher business risk. These ‘higher risk’ items are also scheduled for testing as early as possible in the lifecycle. This approach is complimentary to any development lifecycle from very structured approaches (such as V-Model) through to the more dynamic rapid development methodologies (such as RAD).

6 References

1. Ståhle Amland , Risk-Based Testing and Metrics EuroSTAR99 Proceedings, 1999. http://www.amland.no/articles

2. James Bach on Risk-Based Testing Software Testing and Quality Engineering,Nov/Dec 1999.

3. Cem Kaner, James Bach, Bret Pettichord , “Lessons learned in Software Testing” Wiley, 2001.

4. Shari Lawrence Pfleeger: Risky Business: “What we have yet to learn about SW risk Management” Journal of Systems and Software, 11/2000.

5. Dale Walter Karolak, “Software Engineering Risk Management”, IEEE Computer Society Press, 1996.

6. Norman E. Fenton & Shari Lawrence Pfleeger, "Software Metrics, a rigorous & practical approach", 2nd edition, International Thomson Computer Press, 1997.

7. Linda H. Rosenberg, Ruth Stapko, Albert Gallo , “Risk-based Object Oriented Testing”

**********End of Paper**********

What is the difference between Simultaneous & Concurrent users ?

These are the 2 terms which is often used in Performance Testing.

Simultaneous users are the users who have a valid session in the server. Each of the users would be performing different actions like one doing login , other viewing the reports, etc.

Concurrent users are the users who have a valid session in the server & they are performing the same operation at any point of time. All the concurrent users would perform login, reports download, etc at the same point of time.

Hence for a web site, the number of simultaneous users would be always greater than the concurrent users.For example , a banking web site has the workload of about 30000 simultaneous users & 5000 concurrent users.

Knowing the correct figures of simultaneous & concurrent users is very vital in order to perform performance testing of an application.