Myths about Performance Testing versus Performance Engineering Unveiled

As a software professional, we all have a very good unified common understanding of what is Engineering & Testing team skillset and how different both are. But many of us have difference of opinion when it comes to Performance Testing versus Performance Engineering.

Though Performance Testing comes under the umbrella of Testing, in many aspects Performance testing is very different from the usual functional testing stuff. Be it the effort estimation strategy, test planning , the defect management cycle or the tools knowledge requirement, performance testing is quite different. From test management perspective, there are quite a lot of differences that needs to be exhibited in the management style as well.

Performance Testing is not a type of automation testing where test scripts are created using a tool & automated test run is scheduled. In functional or automation testing, test coverage becomes very important factor whereas in Performance testing, test accuracy becomes essential. Realistic simulation of end user access patterns both quantitatively & qualitatively is a key factor towards successful performance testing but unfortunately this is not measured or expressed using a metric. This has led to a state where anyone who knows using a performance testing tool can do performance testing.

What is Performance Testing

So finally, lets come to the definition, Performance Testing is a type of testing that simulates the realistic end user load & access pattern in a controlled environment in order to identify the responsiveness, speed & stability of the system. It usually requires reporting of system performance metrics like transaction response time, concurrent user loads supported, server throughput, etc  along with additional metrics reporting the software & hardware layer specific performance metrics like browser performance, code performance, server resource utilization,etc  that helps in analyzing the potential bottlenecks that affects the system performance & scalability.

So, yes Performance tester should know how to report the performance metrics of different layers while conducting a performance test. In a 3 tier architecture, performance of individual tiers - web server, application server & DB server , client-side / browser performance, network performance, server hardware performance, etc  needs to be measured & reported. This cannot be considered as an Engineering activity. Deep dive analysis of why a layer specific performance metrics doesn’t meet the SLA can be considered as an Engineering activity.

Usually the confusion starts when it comes to performance bottleneck analysis. I agree there is a thin boundary line exists. Whether it’s a job of performance tester or engineer. Here is my point of view on this topic – Sophisticated performance monitoring tools & Application Performance Management (APM) tools are used independently or integrated with performance testing tools itself to measure & monitor the performance of various tiers (in software layer) & the infrastructure server resource utilizations (in hardware layer) with clearly reported metrics. Hence, it’s the responsibility of a Performance Tester to measure & monitor the performance of end to end system during performance tests & report the observations & findings. The basic straight forward analysis & experience based analysis can be performed by a performance tester to reconfirm the performance problems & observations. Now if the findings requires deep dive analysis, say a specific transaction or method reported to have high response time or a server resource is over utilized, that needs to be debugged further to fine tune it will be the responsibility of Performance Engineer. Application capacity planning, performance modelling & prediction analysis, infrastructure sizing analysis, etc are also core responsibilities of a Performance Engineer. Measuring & Monitoring serveral parameters that can impact overall system performance will be the responsibility of the Performance tester.

What is Performance Engineering

Lets start with the definition for Performance Engineering, it is a discipline that involves systematic practices, techniques & activities during each phase od Software development life cycle (SDLC) to meet  performance requirements. It strives to build performance standards by focusing on the architecture, design & implementation choices.

Hence, Performance needs to be proactively thought throughout the software development phases right from the requirements phase. In this proactive mode,  a Performance Engineer is involved right from the initial SDLC phase to ensure system is build with performance standards. There are several techniques available to validate performance at each SDLC stage even when a testable system is not available.

In reactive mode, when a system is tested for its performance & found to be not scalable, i.e doesn’t meet the non-functional requirements related to response time SLAs, user scalability levels, etc, then a Performance Engineer usually tries to understand the metrics reported by Performance tester & perform a deep dive analysis on the specific layer(s) which doesn’t meet the SLAs. In this case, depending upon the bottleneck reported, specific SMEs like DB architect, Websphere specialist, Network Engineer, etc  can be involved to analyze the performance issue in detail to provide tuning recommendation.

Engineering career path for successful Performance Testers

Performance Testers after gaining good performance testing experience, who possess great interest towards problem analysis & tuning end up having their career path into Performance Engineering.  They are usually not specific technology experts rather they have good understanding of what to be tuned on what circumstances & seem to have good knowledge on various parameters that have to be looked & tuned to have performance & scalability.

These Engineers ususlly have the below skills :

• ·    Possess good experience in reviewing system architecture / deployment architecture & providing suggestions for better performance
• ·   Good knowledge in developing strategies for assuring web / mobile application performance throughout SDLC phases
• ·    Good experience in various Performance testing tools like HP LoadRunner, Jmeter, NeoLoad, etc.
• ·     Good experience in measuring/monitoring performance of various layers involved in end to end system.
• ·     Experience in analysing the application traffic patterns using tools like Omniture, DeepLogAnalyzer, etc.
• ·      Experience in performance monitoring tools & APM tools like Perfmon, HP Sitescope, CA Introscope, Dynatrace, AppDynamics, etc.
• ·         Good experience in using profiling tools like Jprofiler, Jprobe,Jconsole, VisualVM, HP Diagnostics, etc, including GC / JVM analysis tools & heap/thread dump analysis tools.
• ·         Experience in DB profiling tools like Statspack / AWR  / SQL profiler,etc.
• ·         Experience in front end web performance analysis using Yslow, WebPageTest, Ajax Dynatrace, PageSpeed, etc.
• ·         Experience in performance prediction / modelling analysis during early SDLC phases.
• ·         Experience in Capacity planning/sizing through Queuing Theory principles & tools like       TeamQuest, Metron Athene, etc.

A Person with above kind of skillset can also be called Performance Engineers not necessarily they need to have core development skills.  Also, every Performance Engineer might not have skills across all technologies. Based on their practical experience & technical exposure , though they might tend to have knowledge in specific technology, they would carry better understanding of what can make your system scalable & highly available.

A successful Performance Center of Excellence(PCOE) should have engineers with the above qualities. They would have better confidence to guide towards performance assurance rather than people who knows how to execute performance tests using a tool. My sincere advise would be don’t call the COE as testing only or Engineering only, because it will look incomplete from customer point of view. Let your customer’s problem statements drive the project to do performance testing or engineering services. A successful PCOE should comprise of both performance testers & performance engineers complimenting each other with their skillset. 

Looking from customer standpoint, their online business application needs to have complaince against their Performance NFRs (Non-Functional Requirements). To ensure this, as a Performance SME, you need to do testing to measure the performance metrics & certify the system for performance & scalability followed by deep dive analysis & tuning incase (only if) performance SLAs are not met. Unless, your COE have capabilities to do both, your COE will not look complete from customer point of view.

Tips for organizations for setting up a successful Performance Assurance Services is discussed in another post.

Journey towards Performance Analytics & Prediction Models

With Agile & DevOps becoming more & more predominant in software development methodologies, early performance analysis & predictive performance is becoming a norm for business critical & high traffic applications. Performance modelling & prediction analytics using the historical statistics gathered across several layers & that can help in analyzing several what-if scenarios & making quick performance judgement without actually testing the system.

So what is Performance Prediction?

It’s the process of modelling the system performance against simulated user loads by using mathematical expressions. Predictive models can only FORECAST, cannot ascertain what might happen in future as it is probabilistic in nature.

A Performance model uses specific number of building blocks to predict how performance will vary under different what-if scenarios like varied set of load conditions, change in workloads, and change in server capacity, etc.Usually the inputs to the model are expressed in mathematical quantities such as number of users, arrival rate, response time, throughput, resource utilization, etc.

How is Performance Analytics & Prediction correlated ?

A robust performance analytics solution should comprise of system performance data collected from production environment like server resource utilization monitoring, server health & performance metrics from APM tools, etc. A sophisticated analytics solution should have data collected at various stages of software development life cycle phases that can be correlated well using modelling & prediction techniques to forecast the system performance for great combination of what-if scenarios.

When the analytics solution build facilitating different types of data collection & storage used with the sound intelligence built through sound modelling & forecasting algorithms, it becomes predictive analytics which provides realistic forecasts on system performance for what-if scenarios.

There are 2 key types of models – Predictive & Prescriptive. A Predictive model built along with the intelligence to prescribe an action to business to act upon along with the feedback system that tracks outcome produced by the action taken becomes Prescriptive model. For example, a predictive model can predict the peak traffic throughput of the application under test whereas a prescriptive model can predict &recommend / alert business about the need to bring down the resident time of specific layer/method or to upgrade the specific hardware resource with high service demand to meet the performance SLAs with clear data points about the expected performance improvements.

Visualizing the required data as onion layers, the performance prediction accuracy increases when the data layers used for building the analytics solution increases. Some of the major data layers to be considered include performance modelling results & actual performance test results from controlled performance test environment, Network performance simulation results & device side performance metrics (for mobile applications), Test versus Production environment capacity differences , Production infrastructure monitoring statistics,website end user traffic patterns, & web (browser) performance statistics. At least to start with, production environment monitoring statistics & website user traffic statistics data layers are essential ones to do forecasts based on  historical data analysis using regression techniques.

                      

There are several open source & commercial tools generally used to perform the testing & analysis at each of the data layers represented in the figure. The key challenge lies in building the intelligence to parse the results produced by variety of tools & provision a tool agnostic generic reporting dashboard  that can be fed to the prediction / modelling algorithms to support in performance forecasting.

Details on modelling / prediction techniques will be available in a different post.