Myths about Performance Testing versus Performance Engineering Unveiled

As a software professional, we all have a very good unified common understanding of what is Engineering & Testing team skillset and how different both are. But many of us have difference of opinion when it comes to Performance Testing versus Performance Engineering.

Though Performance Testing comes under the umbrella of Testing, in many aspects Performance testing is very different from the usual functional testing stuff. Be it the effort estimation strategy, test planning , the defect management cycle or the tools knowledge requirement, performance testing is quite different. From test management perspective, there are quite a lot of differences that needs to be exhibited in the management style as well.

Performance Testing is not a type of automation testing where test scripts are created using a tool & automated test run is scheduled. In functional or automation testing, test coverage becomes very important factor whereas in Performance testing, test accuracy becomes essential. Realistic simulation of end user access patterns both quantitatively & qualitatively is a key factor towards successful performance testing but unfortunately this is not measured or expressed using a metric. This has led to a state where anyone who knows using a performance testing tool can do performance testing.

What is Performance Testing

So finally, lets come to the definition, Performance Testing is a type of testing that simulates the realistic end user load & access pattern in a controlled environment in order to identify the responsiveness, speed & stability of the system. It usually requires reporting of system performance metrics like transaction response time, concurrent user loads supported, server throughput, etc  along with additional metrics reporting the software & hardware layer specific performance metrics like browser performance, code performance, server resource utilization,etc  that helps in analyzing the potential bottlenecks that affects the system performance & scalability.

So, yes Performance tester should know how to report the performance metrics of different layers while conducting a performance test. In a 3 tier architecture, performance of individual tiers - web server, application server & DB server , client-side / browser performance, network performance, server hardware performance, etc  needs to be measured & reported. This cannot be considered as an Engineering activity. Deep dive analysis of why a layer specific performance metrics doesn’t meet the SLA can be considered as an Engineering activity.

Usually the confusion starts when it comes to performance bottleneck analysis. I agree there is a thin boundary line exists. Whether it’s a job of performance tester or engineer. Here is my point of view on this topic – Sophisticated performance monitoring tools & Application Performance Management (APM) tools are used independently or integrated with performance testing tools itself to measure & monitor the performance of various tiers (in software layer) & the infrastructure server resource utilizations (in hardware layer) with clearly reported metrics. Hence, it’s the responsibility of a Performance Tester to measure & monitor the performance of end to end system during performance tests & report the observations & findings. The basic straight forward analysis & experience based analysis can be performed by a performance tester to reconfirm the performance problems & observations. Now if the findings requires deep dive analysis, say a specific transaction or method reported to have high response time or a server resource is over utilized, that needs to be debugged further to fine tune it will be the responsibility of Performance Engineer. Application capacity planning, performance modelling & prediction analysis, infrastructure sizing analysis, etc are also core responsibilities of a Performance Engineer. Measuring & Monitoring serveral parameters that can impact overall system performance will be the responsibility of the Performance tester.

What is Performance Engineering

Lets start with the definition for Performance Engineering, it is a discipline that involves systematic practices, techniques & activities during each phase od Software development life cycle (SDLC) to meet  performance requirements. It strives to build performance standards by focusing on the architecture, design & implementation choices.

Hence, Performance needs to be proactively thought throughout the software development phases right from the requirements phase. In this proactive mode,  a Performance Engineer is involved right from the initial SDLC phase to ensure system is build with performance standards. There are several techniques available to validate performance at each SDLC stage even when a testable system is not available.

In reactive mode, when a system is tested for its performance & found to be not scalable, i.e doesn’t meet the non-functional requirements related to response time SLAs, user scalability levels, etc, then a Performance Engineer usually tries to understand the metrics reported by Performance tester & perform a deep dive analysis on the specific layer(s) which doesn’t meet the SLAs. In this case, depending upon the bottleneck reported, specific SMEs like DB architect, Websphere specialist, Network Engineer, etc  can be involved to analyze the performance issue in detail to provide tuning recommendation.

Engineering career path for successful Performance Testers

Performance Testers after gaining good performance testing experience, who possess great interest towards problem analysis & tuning end up having their career path into Performance Engineering.  They are usually not specific technology experts rather they have good understanding of what to be tuned on what circumstances & seem to have good knowledge on various parameters that have to be looked & tuned to have performance & scalability.

These Engineers ususlly have the below skills :

• ·    Possess good experience in reviewing system architecture / deployment architecture & providing suggestions for better performance
• ·   Good knowledge in developing strategies for assuring web / mobile application performance throughout SDLC phases
• ·    Good experience in various Performance testing tools like HP LoadRunner, Jmeter, NeoLoad, etc.
• ·     Good experience in measuring/monitoring performance of various layers involved in end to end system.
• ·     Experience in analysing the application traffic patterns using tools like Omniture, DeepLogAnalyzer, etc.
• ·      Experience in performance monitoring tools & APM tools like Perfmon, HP Sitescope, CA Introscope, Dynatrace, AppDynamics, etc.
• ·         Good experience in using profiling tools like Jprofiler, Jprobe,Jconsole, VisualVM, HP Diagnostics, etc, including GC / JVM analysis tools & heap/thread dump analysis tools.
• ·         Experience in DB profiling tools like Statspack / AWR  / SQL profiler,etc.
• ·         Experience in front end web performance analysis using Yslow, WebPageTest, Ajax Dynatrace, PageSpeed, etc.
• ·         Experience in performance prediction / modelling analysis during early SDLC phases.
• ·         Experience in Capacity planning/sizing through Queuing Theory principles & tools like       TeamQuest, Metron Athene, etc.

A Person with above kind of skillset can also be called Performance Engineers not necessarily they need to have core development skills.  Also, every Performance Engineer might not have skills across all technologies. Based on their practical experience & technical exposure , though they might tend to have knowledge in specific technology, they would carry better understanding of what can make your system scalable & highly available.

A successful Performance Center of Excellence(PCOE) should have engineers with the above qualities. They would have better confidence to guide towards performance assurance rather than people who knows how to execute performance tests using a tool. My sincere advise would be don’t call the COE as testing only or Engineering only, because it will look incomplete from customer point of view. Let your customer’s problem statements drive the project to do performance testing or engineering services. A successful PCOE should comprise of both performance testers & performance engineers complimenting each other with their skillset. 

Looking from customer standpoint, their online business application needs to have complaince against their Performance NFRs (Non-Functional Requirements). To ensure this, as a Performance SME, you need to do testing to measure the performance metrics & certify the system for performance & scalability followed by deep dive analysis & tuning incase (only if) performance SLAs are not met. Unless, your COE have capabilities to do both, your COE will not look complete from customer point of view.

Tips for organizations for setting up a successful Performance Assurance Services is discussed in another post.