Myths about Performance Testing versus Performance Engineering Unveiled

As a software professional, we all have a very good unified common understanding of what is Engineering & Testing team skillset and how different both are. But many of us have difference of opinion when it comes to Performance Testing versus Performance Engineering.

Though Performance Testing comes under the umbrella of Testing, in many aspects Performance testing is very different from the usual functional testing stuff. Be it the effort estimation strategy, test planning , the defect management cycle or the tools knowledge requirement, performance testing is quite different. From test management perspective, there are quite a lot of differences that needs to be exhibited in the management style as well.

Performance Testing is not a type of automation testing where test scripts are created using a tool & automated test run is scheduled. In functional or automation testing, test coverage becomes very important factor whereas in Performance testing, test accuracy becomes essential. Realistic simulation of end user access patterns both quantitatively & qualitatively is a key factor towards successful performance testing but unfortunately this is not measured or expressed using a metric. This has led to a state where anyone who knows using a performance testing tool can do performance testing.

What is Performance Testing

So finally, lets come to the definition, Performance Testing is a type of testing that simulates the realistic end user load & access pattern in a controlled environment in order to identify the responsiveness, speed & stability of the system. It usually requires reporting of system performance metrics like transaction response time, concurrent user loads supported, server throughput, etc  along with additional metrics reporting the software & hardware layer specific performance metrics like browser performance, code performance, server resource utilization,etc  that helps in analyzing the potential bottlenecks that affects the system performance & scalability.

So, yes Performance tester should know how to report the performance metrics of different layers while conducting a performance test. In a 3 tier architecture, performance of individual tiers - web server, application server & DB server , client-side / browser performance, network performance, server hardware performance, etc  needs to be measured & reported. This cannot be considered as an Engineering activity. Deep dive analysis of why a layer specific performance metrics doesn’t meet the SLA can be considered as an Engineering activity.

Usually the confusion starts when it comes to performance bottleneck analysis. I agree there is a thin boundary line exists. Whether it’s a job of performance tester or engineer. Here is my point of view on this topic – Sophisticated performance monitoring tools & Application Performance Management (APM) tools are used independently or integrated with performance testing tools itself to measure & monitor the performance of various tiers (in software layer) & the infrastructure server resource utilizations (in hardware layer) with clearly reported metrics. Hence, it’s the responsibility of a Performance Tester to measure & monitor the performance of end to end system during performance tests & report the observations & findings. The basic straight forward analysis & experience based analysis can be performed by a performance tester to reconfirm the performance problems & observations. Now if the findings requires deep dive analysis, say a specific transaction or method reported to have high response time or a server resource is over utilized, that needs to be debugged further to fine tune it will be the responsibility of Performance Engineer. Application capacity planning, performance modelling & prediction analysis, infrastructure sizing analysis, etc are also core responsibilities of a Performance Engineer. Measuring & Monitoring serveral parameters that can impact overall system performance will be the responsibility of the Performance tester.

What is Performance Engineering

Lets start with the definition for Performance Engineering, it is a discipline that involves systematic practices, techniques & activities during each phase od Software development life cycle (SDLC) to meet  performance requirements. It strives to build performance standards by focusing on the architecture, design & implementation choices.

Hence, Performance needs to be proactively thought throughout the software development phases right from the requirements phase. In this proactive mode,  a Performance Engineer is involved right from the initial SDLC phase to ensure system is build with performance standards. There are several techniques available to validate performance at each SDLC stage even when a testable system is not available.

In reactive mode, when a system is tested for its performance & found to be not scalable, i.e doesn’t meet the non-functional requirements related to response time SLAs, user scalability levels, etc, then a Performance Engineer usually tries to understand the metrics reported by Performance tester & perform a deep dive analysis on the specific layer(s) which doesn’t meet the SLAs. In this case, depending upon the bottleneck reported, specific SMEs like DB architect, Websphere specialist, Network Engineer, etc  can be involved to analyze the performance issue in detail to provide tuning recommendation.

Engineering career path for successful Performance Testers

Performance Testers after gaining good performance testing experience, who possess great interest towards problem analysis & tuning end up having their career path into Performance Engineering.  They are usually not specific technology experts rather they have good understanding of what to be tuned on what circumstances & seem to have good knowledge on various parameters that have to be looked & tuned to have performance & scalability.

These Engineers ususlly have the below skills :

• ·    Possess good experience in reviewing system architecture / deployment architecture & providing suggestions for better performance
• ·   Good knowledge in developing strategies for assuring web / mobile application performance throughout SDLC phases
• ·    Good experience in various Performance testing tools like HP LoadRunner, Jmeter, NeoLoad, etc.
• ·     Good experience in measuring/monitoring performance of various layers involved in end to end system.
• ·     Experience in analysing the application traffic patterns using tools like Omniture, DeepLogAnalyzer, etc.
• ·      Experience in performance monitoring tools & APM tools like Perfmon, HP Sitescope, CA Introscope, Dynatrace, AppDynamics, etc.
• ·         Good experience in using profiling tools like Jprofiler, Jprobe,Jconsole, VisualVM, HP Diagnostics, etc, including GC / JVM analysis tools & heap/thread dump analysis tools.
• ·         Experience in DB profiling tools like Statspack / AWR  / SQL profiler,etc.
• ·         Experience in front end web performance analysis using Yslow, WebPageTest, Ajax Dynatrace, PageSpeed, etc.
• ·         Experience in performance prediction / modelling analysis during early SDLC phases.
• ·         Experience in Capacity planning/sizing through Queuing Theory principles & tools like       TeamQuest, Metron Athene, etc.

A Person with above kind of skillset can also be called Performance Engineers not necessarily they need to have core development skills.  Also, every Performance Engineer might not have skills across all technologies. Based on their practical experience & technical exposure , though they might tend to have knowledge in specific technology, they would carry better understanding of what can make your system scalable & highly available.

A successful Performance Center of Excellence(PCOE) should have engineers with the above qualities. They would have better confidence to guide towards performance assurance rather than people who knows how to execute performance tests using a tool. My sincere advise would be don’t call the COE as testing only or Engineering only, because it will look incomplete from customer point of view. Let your customer’s problem statements drive the project to do performance testing or engineering services. A successful PCOE should comprise of both performance testers & performance engineers complimenting each other with their skillset. 

Looking from customer standpoint, their online business application needs to have complaince against their Performance NFRs (Non-Functional Requirements). To ensure this, as a Performance SME, you need to do testing to measure the performance metrics & certify the system for performance & scalability followed by deep dive analysis & tuning incase (only if) performance SLAs are not met. Unless, your COE have capabilities to do both, your COE will not look complete from customer point of view.

Tips for organizations for setting up a successful Performance Assurance Services is discussed in another post.

Journey towards Performance Analytics & Prediction Models

With Agile & DevOps becoming more & more predominant in software development methodologies, early performance analysis & predictive performance is becoming a norm for business critical & high traffic applications. Performance modelling & prediction analytics using the historical statistics gathered across several layers & that can help in analyzing several what-if scenarios & making quick performance judgement without actually testing the system.

So what is Performance Prediction?

It’s the process of modelling the system performance against simulated user loads by using mathematical expressions. Predictive models can only FORECAST, cannot ascertain what might happen in future as it is probabilistic in nature.

A Performance model uses specific number of building blocks to predict how performance will vary under different what-if scenarios like varied set of load conditions, change in workloads, and change in server capacity, etc.Usually the inputs to the model are expressed in mathematical quantities such as number of users, arrival rate, response time, throughput, resource utilization, etc.

How is Performance Analytics & Prediction correlated ?

A robust performance analytics solution should comprise of system performance data collected from production environment like server resource utilization monitoring, server health & performance metrics from APM tools, etc. A sophisticated analytics solution should have data collected at various stages of software development life cycle phases that can be correlated well using modelling & prediction techniques to forecast the system performance for great combination of what-if scenarios.

When the analytics solution build facilitating different types of data collection & storage used with the sound intelligence built through sound modelling & forecasting algorithms, it becomes predictive analytics which provides realistic forecasts on system performance for what-if scenarios.

There are 2 key types of models – Predictive & Prescriptive. A Predictive model built along with the intelligence to prescribe an action to business to act upon along with the feedback system that tracks outcome produced by the action taken becomes Prescriptive model. For example, a predictive model can predict the peak traffic throughput of the application under test whereas a prescriptive model can predict &recommend / alert business about the need to bring down the resident time of specific layer/method or to upgrade the specific hardware resource with high service demand to meet the performance SLAs with clear data points about the expected performance improvements.

Visualizing the required data as onion layers, the performance prediction accuracy increases when the data layers used for building the analytics solution increases. Some of the major data layers to be considered include performance modelling results & actual performance test results from controlled performance test environment, Network performance simulation results & device side performance metrics (for mobile applications), Test versus Production environment capacity differences , Production infrastructure monitoring statistics,website end user traffic patterns, & web (browser) performance statistics. At least to start with, production environment monitoring statistics & website user traffic statistics data layers are essential ones to do forecasts based on  historical data analysis using regression techniques.

                      

There are several open source & commercial tools generally used to perform the testing & analysis at each of the data layers represented in the figure. The key challenge lies in building the intelligence to parse the results produced by variety of tools & provision a tool agnostic generic reporting dashboard  that can be fed to the prediction / modelling algorithms to support in performance forecasting.

Details on modelling / prediction techniques will be available in a different post. 

Upsurge in Savvy Performance Analysis Tools leading to Enigmatic Trend

This article is also published in WOMEN TESTERS – APRIL 2016 EDITION
http://www.womentesters.com/2016/04/

Introduction

With the onset of SMAC (Social, Media, Analytics and Cloud) technologies& Digital transformations, there is an exponential increase in the level of knowledge expected from Performance Testers & Engineers. There has been a sudden increase in the number of vendors providing various application performance management tools to satisfy the growing demand on quick performance insights as high performance is one of the key factors driving the technological revolution. As Performance Engineering focuses on building systems with high performance and performing tuning & optimization to meet the performance, scalability & capacity demands, Performance Testers are usually seen inclined to have their career path towards Performance Engineering. 

This emerging trend in the rise of various performance testing &engineering toolsets for quick performance analysis &easy bottleneck detection has led to a confusing trend withinPerformance Testing/Engineering community.  The upcoming millennials seem to have developed more interest to get their hands on various tools rather than focusing on deepening their concepts &developing thought process for systematic performance problem diagnosis &tuning. This trend is creating a very big challenge for Performance COE’s (Center Of Excellence) as they strive towards successful project delivery in environments where these savvy tools don’t exists.

Tools are just Enablers; Don’t forget to add your Intelligence

The primary skills for Performance Testers is to design & execute various types of performance tests using variety of tools like HP Load Runner or JMeter. Performance Engineers through their vast performance testing experience for years or from developmental background have built stronger credibility as they understand the problems in application performance management including building scalable architecture solutions, selecting right design patterns that enhances performance, code profiling, etc. slightly better & know the areas to be analysed to identify the root cause of the problems when a performance SLA is not met.

Even now, a good performance tester is expected to have HP LoadRunner tool certification. There are no tool agnostic courses for acquiring performance testing capability. Due to which performance testers are usually seem biased & inclined towards specific tool terminologies & concepts. Majority of them think Performance testing is all about developing test scripts for the requested business use cases and create test scenario in the tool to execute load tests for the expected load conditions.

Performance Testers should firstly know how to strategize the tests, it’s not about just conducting load test or stress test using a tool.The key value adds in performance testing on how to bring in better accuracy & real time end user simulation in the performance tests, etc. Your favourite performance testing tool do not have the intelligence to report if the workload (use cases & load distribution) used for the test is incorrectly modelled. Many testers seem to be very comfortable with the knowledge of various tool specific settings without knowing the fact that a wrongly configured think time can end up running a completely wrong testnot even worth doing performance testing for your application.Majority of the production failure RCAs (root cause analysis) revealed improper & inaccurate performance tests due to incorrect design of the workload or improper NFRs used during validation tests.

The Performance testers should think beyond the performance testing tool & focus on gaining stronger understanding of the underlying Performance Testing & Engineering concepts to be able to run accurate & meaningful performance tests.

In recent past, there has been a sudden rise in the popularity of various commercial APM (Application Performance Monitoring) tools like Dynatrace, AppDynamics, New Relic, etc., while these tools have helped in quick & easy performance problem detection providing productivity boom due to lot of performance problem insights & readymade analysis recommendations in few button clicks, it is unfortunate that it has created a situation where Performance Testers /Engineers are more biased to gain experience on these tools rather than understanding the problem analysis concepts & principles.

The Performance Engineers we could find these days also have got biased to tool terminologies and tool provided metrics losing the big picture on overall application performance management & 360 degree of problem analysis. This lack of understanding in some cases have caused great embarrassments during project delivery. But I must agree the fact that these new generation tools are a great tribute for Performance Engineers & have brought our performance bottleneck analysis efforts from weeks to days thereby giving us room to fix it up quickly before it impacts the end user.

The bottom line fact is that there is no substitute for understanding of processes & Performance testing & Engineering concepts/principles which is required for managing performance throughout the SDLC life cycle phases. Gaining knowledge of the tools is important but not to be considered as a substitute for all the underlying performance testing/engineering principles. 

Tools are not an evil. They are great weapons to address the problem quickly. It’s just our perception to use them as enablers needs to be improved & we need to continue focusing on underlying Performance Testing/Engineering principles.

Use of RACI matrix to clarify your Performance Assessment Scope

Recently I had a performance assessment capability (& high level proposal) walkthrough discussion with one of our clients. I had several queries regarding the performance analysis activities scoped in for the engagement. Some of them are shared below:

• Who will perform the analysis on the ground?
• You will pinpoint there is a performance degradation issue & from there on is it our responsibility to analyze and fix it?
• My previous vendor helped me only to run the performance tests whereas its our sole responsibility to do any analysis/tuning to meet the performance SLAs ?
• You are quoting 45 person days effort for performance assessment whereas my current vendor did this using 1/3rd of your quoted efforts ?

I parked all those questions saying will revert back while doing walkthrough of the RACI Matrix slide.

When I moved on RACI Matrix slide, all I said is below :

Performance Testing scope of work will always have below characteristics

CLIENT SMEs are Accountable for ensuring application performance
PERFORMANCE TEAM is Responsible only for carrying out / execution of performance tests
CLIENT SMEs are Accountable & Consulted for performance bottleneck analysis and performance team supports by running retests to verify the performance issues are fixed.
PERFORMANCE TEAM is Informed once the performance fix ready for them to carry out retests.

whereas Performance Engineering scope will have the below characteristics

PERFORMANCE TEAM is Responsible for carrying out / execution of performance tests
PERFORMANCE TEAM is Accountable for ensuring the application performance. (Carryout all possible required analysis to meet the performance SLAs set by the business).
CLIENT SMEs are usually Informed & sometimes Consulted (on need basis for some key decision making) during the detailed performance bottleneck analysis.

Now, finally its your choice to demand what to expect from the performance excellence team.

I personally feel its our (Performance Excellence Team's) responsibility to bring awareness among all our stakeholders on these stuff.

Performance benchmarking assignments would have only subset of activities that is quoted as part of Performance testing assignment.

Performance testing assignment would have only a subset of activities that is quoted as part of performance engineering assignment.

As you move up in the ladder (Performance Benchmarking --> Performance Testing --> Performance Engineering), the scope of activities & effort requirement & thereby cost goes high.

Decide what is required based on the performance criticality of the application and budget availability.

Happy Testing!

Client-Side metrics versus Performance KPIs

Am happy to be back again blogging some useful stuff after a long time !!

Recently when i happened to have a discussion with one of the performance test engineer, I realized to bring this point elaborately looking at his misunderstanding.

The Client-Side performance is of great importance as it assesses end-user experience. It is vital to understand what is happening in the browser & its runtime (JavaScript , DOM). Optimizing the front-end performance from a single user point of view is very vital before testing an application with high user loads. The front-end, client-side performance optimization is quite simple compared to back-end performance optimization where lot of detailed analysis such as code profiling, hardware upgrade, etc. is required.

I experienced it several times & learnt in a hard way that client-side performance is always considered trivial as it is simple and can be quickly performed with right set of tools.

There are quite a good number of tools to do this analysis. To mention a few, Google's WebPageTest tool, Yahoo's YSlow extension, FireFox's FireBug extension, Google's pageSpeed, Ajax Dynatrace for IE, Speed Tracer extension for Google Chrome, etc.

The Key Performance Indicators (KPIs) like Response Time, Throughput, etc cannot be called as client-side metrics. May be you get to see these metrics through the performance test tool, but it is nothing to do with client-side metrics. The KPIs usually represent the system performance after the end of a performance test. These should be called as System Performance metrics or Performance KPIs.