Thanks a lot viewers. Thanks for your encouragement by your kind mails.
I am planning to publish a book on Performance Testing comprising all that i learned during the past 3 years of the experience in my career.
I started my carrer as a functional tester and pushed or pulled into Performance testing fortunately thereby i came to know about this area. Mine is a hard way and i didnt had much support in terms of helping hands to get to know about the fundamendals before i started handling the projects. Slowly i learned a lot and finally ended up in unlearning a lot because of the ignorance i had initial days.
The objective of this book is to help fellow performance testers so that they can spend their quality time in achieving great things rather than reinventing the wheel :)
"Performance Testing Handbook - A Comprehensive Guide for Beginners"
The bird's eye view of topics covered in this book includes,
1. Why Performance Testing is crucial.
2. Dynamics and myths in performance testing
3. Deriving Quantitative performance test goals
4. Common mistakes in performance testing which i learned
5. Overview of performance test tools - How to choose the best tool
6. Setting up the performance test environment
7. Comprehensive support for writting test scripts
8. Application benchamarking
9. Quick look at the Industry standard benchmarks
10. A helping in setting up the performance Test Strategy
11. Performance Test Monitoring
12. Performance Test Execution - When and how to choose various tests (Load, Stress, Volume, Soak, Endurance tests).
13.Bottlenecks identification strategy
14.Reporting - How does it matters so much
15.Road Ahead - towards mathematical modeling and queuing theory basics
Friday, August 10, 2007
Thursday, August 9, 2007
Web Log Analysis
A typical issue which i faced during web log analysis...
While conducting Performance testing for applications which are falling into the category of Post-Production case, the most important activity that needs to be done is web log analysis. The web server logs are the gold mines which hold all the information about the server during the load condition.
During the web log analysis, look for the peak hour traffic time and identify what is the server load in terms of hits/sec and in terms of users. Based on this statistics, derive performance test goals by considering the business forecasts.
I would like to share a typical issue which I faced while doing web log analysis for a typical post production category of an application. The application under test uses a shared web-app server (IIS) in which there was 2 other applications deployed. Now, the problem is the web server log files have the traces of all 3 applications together as these files are common for a server and not specific for each of the applications running on the server.
The web log analyzer tool which I was using doesn’t have any option to set filters. In that case, I tried using a web log analyzer tool (123LogAnalyzer tool) which provides a filter option to filter out the traces pertaining to a specific application and export the filtered log file into a text file. For example, if the application under test has a context root as ‘methodologies’ (http://132.23.34.21/methodologies/......), then try setting up this URL in the filter and create the text files for your application of interest.
Now you could use the newly generated text file to feed it into any of the log analyzer tool which you are confident enough and proceed with your performance goal setting. My kind advice is that before choosing the log analysis tool of your choice, analyze the accuracy of log analysis thoroughly.
For more details on the various web log analysis tools, look for the web log analysis comparison article available in my blog.
Hope it helps!!!
While conducting Performance testing for applications which are falling into the category of Post-Production case, the most important activity that needs to be done is web log analysis. The web server logs are the gold mines which hold all the information about the server during the load condition.
During the web log analysis, look for the peak hour traffic time and identify what is the server load in terms of hits/sec and in terms of users. Based on this statistics, derive performance test goals by considering the business forecasts.
I would like to share a typical issue which I faced while doing web log analysis for a typical post production category of an application. The application under test uses a shared web-app server (IIS) in which there was 2 other applications deployed. Now, the problem is the web server log files have the traces of all 3 applications together as these files are common for a server and not specific for each of the applications running on the server.
The web log analyzer tool which I was using doesn’t have any option to set filters. In that case, I tried using a web log analyzer tool (123LogAnalyzer tool) which provides a filter option to filter out the traces pertaining to a specific application and export the filtered log file into a text file. For example, if the application under test has a context root as ‘methodologies’ (http://132.23.34.21/methodologies/......), then try setting up this URL in the filter and create the text files for your application of interest.
Now you could use the newly generated text file to feed it into any of the log analyzer tool which you are confident enough and proceed with your performance goal setting. My kind advice is that before choosing the log analysis tool of your choice, analyze the accuracy of log analysis thoroughly.
For more details on the various web log analysis tools, look for the web log analysis comparison article available in my blog.
Hope it helps!!!
Wednesday, July 4, 2007
Proactive Vs Reactive Approach in Performance Testing
Reactive Approach
The Performance Testing activity is often considered as a reactive way of performance management. In most of the cases, the system performance is never thought of during the early phases of Software Development Life Cycle phases (SDLC). Performance is often thought of only as a last activity after the System Testing phase.
And most of the time , the performance bottlenecks revolves around the system architecture or the system design which is a very high cost activity to think about & in certain cases , the system is put into trash because of the huge deviations in the performance constraints.
Basically waiting for the performance problems to appear & then dealing with it at the end is always not a better approach. Mostly Performance Testing is considered as a reactive approach as there is not much importance is given to the system during early life cycle phases. It is more a ‘fix-it-later’ approach which is not that effective.
Proactive Approach
The Proactive approach anticipates the performance problems well in advance & adopts techniques for mitigating them. The Importance of the Performance is thought about from the Initial phase of Software Development Life Cycle (SDLC) & various performance engineering activities are identified for the system.
The disadvantages of ‘fix-it-later’ approach are well understood & engineering practices are adopted to analyze the system design in performance angle.
The Performance Testing activity is often considered as a reactive way of performance management. In most of the cases, the system performance is never thought of during the early phases of Software Development Life Cycle phases (SDLC). Performance is often thought of only as a last activity after the System Testing phase.
And most of the time , the performance bottlenecks revolves around the system architecture or the system design which is a very high cost activity to think about & in certain cases , the system is put into trash because of the huge deviations in the performance constraints.
Basically waiting for the performance problems to appear & then dealing with it at the end is always not a better approach. Mostly Performance Testing is considered as a reactive approach as there is not much importance is given to the system during early life cycle phases. It is more a ‘fix-it-later’ approach which is not that effective.
Proactive Approach
The Proactive approach anticipates the performance problems well in advance & adopts techniques for mitigating them. The Importance of the Performance is thought about from the Initial phase of Software Development Life Cycle (SDLC) & various performance engineering activities are identified for the system.
The disadvantages of ‘fix-it-later’ approach are well understood & engineering practices are adopted to analyze the system design in performance angle.
The integration of performance engineering activities with the SDLC phases is provided in the below table.
Tips for developing effective Performance Test Scripts
The Test scripts are generated to mimic the user behavior during the performance tests. The test scripts enable the performance tester to simulate the real time user behavior by providing the configuration options. The Test scripts needs to be modeled in such a way that it exactly simulates the user actions & user think times.
The Test scripts should have the transaction names or browser requests identified for which the response time & throughput metrics needs to be collected. These requests should be named with proper naming convention so that it becomes easy to refer to a specific transaction by its unique name.
Running the script containing the right set of transactions alone doesn’t mimic the real time user behavior. Each transaction should be always coupled with another important metric called as think time, which is the time taken by the user to think and to navigate between various pages. This wait time which is placed between each transaction determines the accuracy of the scripts simulating the user behavior. Analyzing the think time between each transaction is very critical as it determines the load on the server at any point of time. By conducting discussions or surveys with business analysts & end users of the application, the think time for each transaction can be identified. Say, you have 2 transactions in your script, Login transaction followed by the think time of about 10 seconds followed by the Logout transaction. If the think time is changed to 5 seconds, then the frequency of the login & logout requests send to the server increases, hence the load (requests/sec) on the server increases. If the think time is changed to 20 seconds, then the server load (requests/sec) decreases as the requests are sent in 20 seconds interval.
The test data parameterization is another important activity which needs to be taken care in the scripts. Care should be taken so that enough test data is provided in the script so that each test data is not used more than once when the tests are run. For example, the user login credentials needs to be parameterized in the script. If the test is planned to run for 100 users, then the script should have 100 login credentials available in the script. If more than 1 user is allowed to use the same test data, then the response time would be varying as it might be caching the web pages or the session might get reused. It’s always recommended to make sure that the script is supplied with enough test data to be used uniquely for each user. By using parameterization, we can make sure that the correct formatted unique data is sent from the client-side for each user during the course of the test run. From the server side, there might be some dynamic data generated for each user which may be unique for each user. For example the session id of each user would be unique. If the server passes the session id of the user while returning the response for a specific client side request, then this part of the script cannot replay correctly if it is run for a different user id from the parameterized test data.
This kind of server-side dynamic data needs to be handled in the script as well. Sometimes the recorded script cannot replay because of some specific dynamic values which are generated by the server. This kind of dynamic values needs to be first studied whether it’s unique with respect to each logged in user or unique for each run even for the same user. Sometimes the user action gets completed even if the dynamic values are not handled. Hence, first analyze the logic under which there is a dynamic value getting generated from the server end & accordingly handle the dynamic value. Most of the time the dynamic value is handled by identifying the corresponding left & right boundary & capturing the dynamic value, then the captured value can be substituted against the variable which is holding the dynamic value.
Last but not least, make sure the script is very realistic in terms of the user actions. Try to have a single script which covers all the flows of the application & use random number generator function to distribute the user load across various functional flows. Sometimes it’s good to have multiple scripts as long as the distribution of the load across scripts is handled separately.
The Test scripts should have the transaction names or browser requests identified for which the response time & throughput metrics needs to be collected. These requests should be named with proper naming convention so that it becomes easy to refer to a specific transaction by its unique name.
Running the script containing the right set of transactions alone doesn’t mimic the real time user behavior. Each transaction should be always coupled with another important metric called as think time, which is the time taken by the user to think and to navigate between various pages. This wait time which is placed between each transaction determines the accuracy of the scripts simulating the user behavior. Analyzing the think time between each transaction is very critical as it determines the load on the server at any point of time. By conducting discussions or surveys with business analysts & end users of the application, the think time for each transaction can be identified. Say, you have 2 transactions in your script, Login transaction followed by the think time of about 10 seconds followed by the Logout transaction. If the think time is changed to 5 seconds, then the frequency of the login & logout requests send to the server increases, hence the load (requests/sec) on the server increases. If the think time is changed to 20 seconds, then the server load (requests/sec) decreases as the requests are sent in 20 seconds interval.
The test data parameterization is another important activity which needs to be taken care in the scripts. Care should be taken so that enough test data is provided in the script so that each test data is not used more than once when the tests are run. For example, the user login credentials needs to be parameterized in the script. If the test is planned to run for 100 users, then the script should have 100 login credentials available in the script. If more than 1 user is allowed to use the same test data, then the response time would be varying as it might be caching the web pages or the session might get reused. It’s always recommended to make sure that the script is supplied with enough test data to be used uniquely for each user. By using parameterization, we can make sure that the correct formatted unique data is sent from the client-side for each user during the course of the test run. From the server side, there might be some dynamic data generated for each user which may be unique for each user. For example the session id of each user would be unique. If the server passes the session id of the user while returning the response for a specific client side request, then this part of the script cannot replay correctly if it is run for a different user id from the parameterized test data.
This kind of server-side dynamic data needs to be handled in the script as well. Sometimes the recorded script cannot replay because of some specific dynamic values which are generated by the server. This kind of dynamic values needs to be first studied whether it’s unique with respect to each logged in user or unique for each run even for the same user. Sometimes the user action gets completed even if the dynamic values are not handled. Hence, first analyze the logic under which there is a dynamic value getting generated from the server end & accordingly handle the dynamic value. Most of the time the dynamic value is handled by identifying the corresponding left & right boundary & capturing the dynamic value, then the captured value can be substituted against the variable which is holding the dynamic value.
Last but not least, make sure the script is very realistic in terms of the user actions. Try to have a single script which covers all the flows of the application & use random number generator function to distribute the user load across various functional flows. Sometimes it’s good to have multiple scripts as long as the distribution of the load across scripts is handled separately.
Happy Scripting!!
Friday, June 29, 2007
How to run effective performance tests
To plan for an effective performance test, follow the below four important steps
1.The first step is to understand the web site’s actual usage pattern. Mostly this information can be collected from the web server log files which are considered as the ‘gold mines’ to know every possible detail about the server load. The ‘log analyzers’, tools which helps to analyze the web log files and provide the analysis report in terms of number of user sessions, number of unique users, number of errors, peak day traffic, peak hour traffic, user request arrival pattern, frequently accessed pages, user navigation pattern, etc. From this analysis, the actual usage of the server can be identified.
2.The second step is to design the performance test approach. Based on the business forecast of the user load increase, develop service level agreement (SLAs) that has to be met by the application in order to handle the user load in the target environment. The analysis done in the first step helps to set quantitative SLA for the application. Design the test cases that needs to be tested along with the parameters like ramp up strategy, test duration, think time settings, test scenarios to be tested, different load conditions & types of tests along with the pass/fail criteria. Baseline the test approach & use this as the reference for the rest of the activities.
3.The third step is to identify the performance test tool to help us in simulating the user load on the server. Look for performance test tools available in the market (Licensed or freeware tools) based on the budget constraints & technologies supported by the test tool. Using the test tool of choice, develop the test script which simulates the actual user access pattern. Have enough test data available in the test script in order to simulate different users using different data available in the script.
4.The fourth step is to look for the readiness of the test environment. The test environment includes the application’s server infrastructure & also the load test environment. Enough care should be taken to ensure that no other users access the application during the test run & the servers are isolated from any other usage. In the case of load test environment, ensure that load generators are in place to generate the required load, the availability of test tool license to support the load to be simulated, test scripts are developed and ready with the required test data & think time, server monitors are configured with the required performance counters & the server admins are informed for their assistance in case of server failures.
1.The first step is to understand the web site’s actual usage pattern. Mostly this information can be collected from the web server log files which are considered as the ‘gold mines’ to know every possible detail about the server load. The ‘log analyzers’, tools which helps to analyze the web log files and provide the analysis report in terms of number of user sessions, number of unique users, number of errors, peak day traffic, peak hour traffic, user request arrival pattern, frequently accessed pages, user navigation pattern, etc. From this analysis, the actual usage of the server can be identified.
2.The second step is to design the performance test approach. Based on the business forecast of the user load increase, develop service level agreement (SLAs) that has to be met by the application in order to handle the user load in the target environment. The analysis done in the first step helps to set quantitative SLA for the application. Design the test cases that needs to be tested along with the parameters like ramp up strategy, test duration, think time settings, test scenarios to be tested, different load conditions & types of tests along with the pass/fail criteria. Baseline the test approach & use this as the reference for the rest of the activities.
3.The third step is to identify the performance test tool to help us in simulating the user load on the server. Look for performance test tools available in the market (Licensed or freeware tools) based on the budget constraints & technologies supported by the test tool. Using the test tool of choice, develop the test script which simulates the actual user access pattern. Have enough test data available in the test script in order to simulate different users using different data available in the script.
4.The fourth step is to look for the readiness of the test environment. The test environment includes the application’s server infrastructure & also the load test environment. Enough care should be taken to ensure that no other users access the application during the test run & the servers are isolated from any other usage. In the case of load test environment, ensure that load generators are in place to generate the required load, the availability of test tool license to support the load to be simulated, test scripts are developed and ready with the required test data & think time, server monitors are configured with the required performance counters & the server admins are informed for their assistance in case of server failures.
Subscribe to:
Posts (Atom)