Monday, August 8, 2016

Performance Engineering – Dusting out Subjectivity & Bringing light to hidden Track


How can it be possible ? But meaning of Performance Engineer is subjective as of now. Let’s try to break it. Performance is a quality attribute which cannot be achieved in a SDLC phase. For assuring Performance, you need many players in SDLC phases but you can’t call everyone as Performance Engineers. We need architectural / design expert, UX designer, development lead, performance tester, capacity planner, etc. to assure my digital product performance & thereby user experience.

Performance Engineering is a discipline that involves systematic practices, techniques & activities during each phase of Software development life cycle (SDLC) to meet the performance requirements. It strives to build performance standards by focusing on the architecture, design & implementation choices.

Ideally speaking, performance being one of the CTQ (Critical To Quality) attribute, it needs to be proactively thought throughout the software development phases right from the requirements phase till ongoing production maintenance. In this proactive mode, a Performance Development Engineer (who is a solution architect with technology expertise aware of right architectural & design pattern choices for system performance & scalability) should be involved right from the initial SDLC phase to ensure system is build with performance standards.

In a reactive mode, system performance is not thought till end of implementation / functional testing phase leading to reverse engineering of the digital product sometimes leading to architectural/design level changes. In proactive or reactive mode, when a Performance Test Engineer tries to test & assess the developed system for its performance & when SLAs are not met, he performs a deep dive analysis on the specific layer(s) which doesn’t meet the SLAs. In this case, depending upon the type & complexity of the bottleneck being analyzed, deep dive analysis & tuning can be done by self or specialized SMEs like DB architect, Websphere specialist, Network Engineer, etc can be involved to analyze the performance issue in detail to provide tuning recommendation. (Note: I hope we all accept the fact that a Performance Tester should have the capability to test & assess the performance problems in the system at all layers & report RCA findings whereas a Performance Engineer is an experienced person who can be a technology expert with good bottleneck analysis/tuning skills to provide recommendation for fixing the issue. But ideally a sound Performance Tester upon gaining good experience develops the skills of a Performance Engineer).

Gaining Clarity on the Contradict

Here comes the disagreement – If you notice above, in both proactive & reactive modes, a Performance Engineer is involved. But note, I have called the former as Performance Development Engineer & later as Performance Test Engineer. The skills of a Performance Development Engineer can be very different from that of Performance Test Engineer.

But we need to remember, we can have both Performance Development Engineer & Performance Test Engineer available from the early SDLC phases as both are not duplicating the skills, actually they are complimenting each other.  This is very similar to this scenario - Testing done by development team (Unit Testing) & testing done by testing team (System Testing) has its own objectives & advantages. They complement each other & try to find as much defects as early to reduce the cost of fixing the defect.

I look at a Performance Test Engineer to be a Performance Assurance Expert where (s)he need not be a technology expert (building system with performance is the job of a Performance Development Engineer), rather (s)he needs to look at the digital product from all perspective to validate whether the digital product will create great user experience by providing expected performance. 

Apart from the knowledge of various testing, monitoring & APM tools, (s)he need to be aware of technology agnostic performance principles & know how to detect performance issues by strategizing right type of performance tests with right workload model. With thorough performance bottleneck analysis skills across all layers, need to have matured thought-process on when my hardware will saturate/reach its thresholds affecting the scalability (though may not be a capacity planning expert).

Hidden Track of Performance Engineering

Though Performance Engineering is itself very broad, but still there is something that is hidden or forgotten. Hidden/Forgotten I meant, it has not gained so much popularity comparatively & we don’t find people easily with these skills very often.

Being the performance assurance expert, Performance Engineer also needs to be capable of employing scientific/mathematical principles to engineer various test activities (like verifying whether test or workload is valid mathematically, forecast peak traffic hour workload, how to map test results from low end environment to PROD hardware, etc), to perform prediction or performance extrapolation, to bring in mathematical strategies to model & validate the performance even before the system is completely built, etc.

Generally speaking, with respect to Test Competency, every Performance Tester aspires to become a Performance Engineer, who is well versed in bottleneck analysis skills, profiling, tuning, etc. But I have met very few Performance Testers who aspire to gain knowledge on Queuing Theory principles to get into this hidden world of Performance prediction, modeling & application capacity planning. This track is the base for the onset of high end capacity planning tools & recent boom on Performance analytics & Predictive/Prescriptive Modeling on Performance data. Many businesses are having great demand for this skillset.

Performance Test Engineers need to have plans to expand their knowledge on this space for having successful & great learning career.

If you disagree, don’t forget to share your point of view, for the benefit of Performance Testers/Engineers. Together lets strive to create more awareness & remove any subjective usage of 
the terms & hold the torch on hidden tracks.


Happy Performance Testing & Engineering!!

Wednesday, August 3, 2016

Precise Workload Analysis in Performance Testing & Application Capacity Planning – The Secret Sauce




Many Performance Testers/Engineers underestimate the importance analyzing the historical end user access patterns while developing the workload model for performance testing / application capacity planning. On majority of my audits doing RCA on production performance issues, the culprit will be wrong workload. The Performance test strategy talks about various types of tests that will be planned, infrastructure components that will be monitored, type of analysis that be performed, etc but when it comes to workload, it is always expected to be provided by customer or business analysts.  Definitely, our Customer / Business Analysts knows who are end users & frequently used business flows, but its sole responsibility of Performance Tester/Engineer to understand (& sometimes educate customers) the additional detailed analysis required in order to increase the accuracy of our performance tests.

We need to remember the fact that if the workload selected for running the performance tests is not reflective of realistic end user access pattern, the entire test results will go wrong & will result in jeopardy. Remember the below points during your workload analysis:
  •  Analyze your end user access patterns. Try to understand your user’s behaviors.
  •  Identify your average & peak point workloads in your historical trends.
  • Define your peak point workload both quantitatively & qualitatively.
  • During peak point pattern analysis pay attention to ignore outliers.

Now create workload model that needs to be used for your performance tests. You can have more than one workload model for your tests. I mean the workload used for your load test can be different from that of endurance test or stress test. It all depends on your end user access patterns. Remember, knowing basics of Queuing Theory (Operational laws) can help to validate the correctness of your workload model & even to an extent whether your peak hour SLA is valid.

If you are dealing with a very business critical & high availability application where it’s really worth, spend time in understanding the underlying statistical distribution pattern for your peak traffic hour workloads. For a web application accessed by independent geographical distributed users, usually fall to a Poisson distribution or Self-Similar distribution. In simple terms, which distribution does my application workload belong to is about analyzing how much bursts & spikes does my peak hour workload have. Representing the burstiness of your traffic using a metric called Hurst & employing various techniques to quantify the Hurst value will confirm which statistical distribution your application fall into & how much your peak hour workload can vary in future.

For Application Capacity Planning, choosing right workload peak points becomes very essential. Unless you choose a series of peak point workloads from your historical statistics & understand the quantitatively & qualitatively what the workload really comprises of, you will not succeed in accurate forecasting of hardware demands for your application. Applying analytical modeling techniques to answer business demanded what-if scenarios can be made possible using carefully selected workload. Without doing this basic homework, you cannot rightly size your infrastructure for the projected business loads.

Also, most of the capacity planning techniques requires actual application performance benchmarks for careful extrapolation / forecasts. Performance benchmarking becomes very important in capacity planning to understand the hardware resource requirements (represented as service demands) & other performance characteristics of your application. Using right workload to carryout performance benchmarking is the first step towards successful application capacity planning.  

Happy Workload Analysis & Modeling!!