Showing posts with label Queuing Theory basics. Show all posts
Showing posts with label Queuing Theory basics. Show all posts

Monday, August 8, 2016

Performance Engineering – Dusting out Subjectivity & Bringing light to hidden Track


How can it be possible ? But meaning of Performance Engineer is subjective as of now. Let’s try to break it. Performance is a quality attribute which cannot be achieved in a SDLC phase. For assuring Performance, you need many players in SDLC phases but you can’t call everyone as Performance Engineers. We need architectural / design expert, UX designer, development lead, performance tester, capacity planner, etc. to assure my digital product performance & thereby user experience.

Performance Engineering is a discipline that involves systematic practices, techniques & activities during each phase of Software development life cycle (SDLC) to meet the performance requirements. It strives to build performance standards by focusing on the architecture, design & implementation choices.

Ideally speaking, performance being one of the CTQ (Critical To Quality) attribute, it needs to be proactively thought throughout the software development phases right from the requirements phase till ongoing production maintenance. In this proactive mode, a Performance Development Engineer (who is a solution architect with technology expertise aware of right architectural & design pattern choices for system performance & scalability) should be involved right from the initial SDLC phase to ensure system is build with performance standards.

In a reactive mode, system performance is not thought till end of implementation / functional testing phase leading to reverse engineering of the digital product sometimes leading to architectural/design level changes. In proactive or reactive mode, when a Performance Test Engineer tries to test & assess the developed system for its performance & when SLAs are not met, he performs a deep dive analysis on the specific layer(s) which doesn’t meet the SLAs. In this case, depending upon the type & complexity of the bottleneck being analyzed, deep dive analysis & tuning can be done by self or specialized SMEs like DB architect, Websphere specialist, Network Engineer, etc can be involved to analyze the performance issue in detail to provide tuning recommendation. (Note: I hope we all accept the fact that a Performance Tester should have the capability to test & assess the performance problems in the system at all layers & report RCA findings whereas a Performance Engineer is an experienced person who can be a technology expert with good bottleneck analysis/tuning skills to provide recommendation for fixing the issue. But ideally a sound Performance Tester upon gaining good experience develops the skills of a Performance Engineer).

Gaining Clarity on the Contradict

Here comes the disagreement – If you notice above, in both proactive & reactive modes, a Performance Engineer is involved. But note, I have called the former as Performance Development Engineer & later as Performance Test Engineer. The skills of a Performance Development Engineer can be very different from that of Performance Test Engineer.

But we need to remember, we can have both Performance Development Engineer & Performance Test Engineer available from the early SDLC phases as both are not duplicating the skills, actually they are complimenting each other.  This is very similar to this scenario - Testing done by development team (Unit Testing) & testing done by testing team (System Testing) has its own objectives & advantages. They complement each other & try to find as much defects as early to reduce the cost of fixing the defect.

I look at a Performance Test Engineer to be a Performance Assurance Expert where (s)he need not be a technology expert (building system with performance is the job of a Performance Development Engineer), rather (s)he needs to look at the digital product from all perspective to validate whether the digital product will create great user experience by providing expected performance. 

Apart from the knowledge of various testing, monitoring & APM tools, (s)he need to be aware of technology agnostic performance principles & know how to detect performance issues by strategizing right type of performance tests with right workload model. With thorough performance bottleneck analysis skills across all layers, need to have matured thought-process on when my hardware will saturate/reach its thresholds affecting the scalability (though may not be a capacity planning expert).

Hidden Track of Performance Engineering

Though Performance Engineering is itself very broad, but still there is something that is hidden or forgotten. Hidden/Forgotten I meant, it has not gained so much popularity comparatively & we don’t find people easily with these skills very often.

Being the performance assurance expert, Performance Engineer also needs to be capable of employing scientific/mathematical principles to engineer various test activities (like verifying whether test or workload is valid mathematically, forecast peak traffic hour workload, how to map test results from low end environment to PROD hardware, etc), to perform prediction or performance extrapolation, to bring in mathematical strategies to model & validate the performance even before the system is completely built, etc.

Generally speaking, with respect to Test Competency, every Performance Tester aspires to become a Performance Engineer, who is well versed in bottleneck analysis skills, profiling, tuning, etc. But I have met very few Performance Testers who aspire to gain knowledge on Queuing Theory principles to get into this hidden world of Performance prediction, modeling & application capacity planning. This track is the base for the onset of high end capacity planning tools & recent boom on Performance analytics & Predictive/Prescriptive Modeling on Performance data. Many businesses are having great demand for this skillset.

Performance Test Engineers need to have plans to expand their knowledge on this space for having successful & great learning career.

If you disagree, don’t forget to share your point of view, for the benefit of Performance Testers/Engineers. Together lets strive to create more awareness & remove any subjective usage of 
the terms & hold the torch on hidden tracks.


Happy Performance Testing & Engineering!!

Thursday, November 27, 2008

My Book on Software Performance Testing for Begineers

Here is the updated table of contents of my book.... I am happy to share that the book will be available in my blog in downdable pdf format within a week. Preview of all the chapters will be available in couple of days.

Software Performance Testing Handbook
A Comprehensive Guide for Beginners

Author : Ramya Ramalinga Moorthy

Contents

Chapter 1: Introduction.. 7
What is Performance Testing.. 7
How Performance Testing is different from Functional Testing.. 8
What is an E-Business Application.. 8
Components of a Web Site.. 9
Why Performance Testing is Crucial.. 10
Performance Testing Jargons.. 10

Chapter 2: Performance Testing – A Quick Overview.. 15
Performance Management.. 15
Performance Testing Approach.. 16
Dynamism in Performance Testing.. 18
Application Performance Management.. 19
Myths about Performance Testing.. 19
Performance Process Maturity Model.. 22

Chapter 3: Deriving Performance Test Goals.. 25
Why to set the Performance Test Goals.. 25
Know your customer.. 25
Deriving Quantitative Goals.. 26
Industrial Standards for Benchmarks.. 27
User behavior Modeling.. 27
Importance of automation in Performance Testing.. 29
Performance Test Estimation.. 29

Chapter 4: Workload Identification.. 33
What is Workload? 33
Types of Workload.. 33
Customer Behavior Modeling.. 36
Customer Behavior Model Graph (CBMG).. 36
User Community Modeling Language (UCML).. 36
Web log Analysis.. 36
Web Log Analysis Tools Overview.. 37
Web Log Analyzers Summary.. 38
Web Log Analysis Metrics.. 42
Web Site Traffic Monitoring.. 46
Overview of Statistical distributions.. 46

Chapter 5: Overview of Performance Testing Tools.. 51
Performance Test Tool Requirements.. 51
Market Tools.. 52
Open Vs Licensed Performance test tools.. 52
Criteria for choosing the Performance test tool.. 52
The most important tool.. 53
Performance Testing Tools Overview.. 53

Chapter 6: Setting up Performance Test Environment and Performance Test Scripts development best practices.. 56
Know the Test and Production environment.. 56
Test environment Isolation.. 56
Network Isolation.. 57
Load Generators.. 57
Test Data Generators.. 58
Test Execution Environment.. 58
How to choose the test scenarios.. 59
Tips for writing successful performance test scripts.. 60
Real time versus Virtual user Mapping.. 63

Chapter 7: Application Benchmarking.. 64
What is benchmarking?.. 64
Why should we benchmark the applications?.. 64
Software Benchmarking.. 64
Hardware Benchmarking.. 65
Industry Standards for Benchmarking.. 65
Transaction processing Performance Council (TPC).. 65
Standard Performance Evaluation Council.. 66
Mistakes in Benchmarking.. 67

Chapter 8: Performance Test Execution 68
Quality of Service of web application.. 68
Virtual User Simulation.. 69
Pre-requisites for Performance Tests.. 69
Types of Performance Tests.. 70
Performance Test Execution Approach.. 73
Tips for Load and Stress Test Execution.. 76
Test Execution Factors.. 77

Chapter 9: Performance Test Monitoring 81Introduction to Performance Monitoring.. 81
What Counters needs to be monitored?.. 81
Key Performance Counters.. 82
Performance Monitoring in Windows platform.. 85
Using Perfmon.. 85
Post production Monitoring.. 87
Benefits of Performance Monitoring.. 87

Chapter 10: Performance Bottleneck Analysis.. 88
Scott’s Recommendation on Scatter Charts.. 88
Scatter Chart Types.. 89
Performance Test Analysis.. 90
Best Practices on Performance Bottleneck Analysis and Isolation.. 92
Performance Testing Vs Profiling.. 94
HP Diagnostics.. 94
SQL Profiler.. 98
Know your Limits.. 99
Typical bottlenecks on DB server.. 100

Chapter 11: Performance Test Reporting.. 102
Why is it Critical.. 102
How to report the results of performance testing.. 102
Components of a good Test Report.. 103
Components of a Good Visual 104
Response Time Reporting 104
Best Practices of Test Reporting recommended by Scott Barber 105

Chapter 12: Road Ahead – Moving towards advanced Performance Engineering techniques.. 107
Moving towards Performance Engineering.. 107
What is Queuing Theory?.. 107
Types of Queuing System.. 108
What can be modeled as Queues?.. 109
Limitations of Queuing Theory.. 110
Operational Laws.. 110
a. Utilization Law.. 110
b. Little’s Law.. 111
c. Interactive Response Time Law.. 112
d. Forced Flow Law.. 112
e. Flow Balance Assumption.. 113
f. Bounding Law of Service Demand.. 113
Capacity Planning.. 114
Why Capacity Planning is required.. 114
When is it required?.. 115

Happy Learning!!!!

Tuesday, March 25, 2008

Response Time Vs Queue Length Vs Server Utilization

1. Is the system Response time directly / inversely proportional to the server utilization ?
2. Will there be queues(for CPU/Memory/Disk) if Utilization is less than 100% ?


Actually speaking, there is no correlation between system response time and the server utilization. The Response time increase might happen even when the server is less utililised. (To illustrate this, look at the ATM example available in the PEA site - http://www.pea-online.com/resources.htm).

Most of us dont understand this relationship. Increase in response time (during high load) is caused only because of queueing of the requests. The user request arrival pattern contributes to the high response time. Due to the adhoc user request arrival pattern, long queues are formed in various service centers which leads to high response time.

Response Time - What are the various components of Response time?

Waiting time (Queuing time) + Processing time = Response Time.

For a server, the processing time is always the same irrespective of load. For example, the server might take 2 seconds of processing during 1 user load and during 1000 users load for a transaction. But the waiting time would be high during 1000 users load, which drastically increases the overall response time. So, response time increases, if the waiting time increases which is caused by long queue.

Also, dont think that there will be queues only if Utilization is more than 100%. There can be queues formed even when the Utilization is less. So, everything boils down to knowing the user arrival pattern of the system. This is going to determine, when the system is going to be loaded and break.

Hence, knowing the user arrival pattern of a system is very important rather than setting mere goals to perform load/stress test. Analysing the user arrival pattern of an application would help in setting realistic goals which becomes more of organization logistics issue and most organizations bypass and pay for it later.


Wednesday, March 21, 2007

Relationship between Metrics

Things to note : Queuing Theory basics

1. The utilization of a service center is dependent on the Arrival rate.
2. The Service Demand remains constant irrespective of the load (for Load Independent devices).
3. Throughput increases with the Utilization.
4. Queues will be formed even when the utilization is less than 100% & its also possible that there can be no queues during 100% utilization too. Hence there is no relationship between the Utilization & Queue length.
5.The response time does not depend on the Utilization. It depends on the Queue length, hence the arrival pattern.
6. The maximum CPU utilization that can be achieved can be calculated using the formula Queue length / Queue Length + 1.
(i.e) For a four CPU m/c maximum utilization that can be achieved = 4/5 = 80%