Journey towards Performance Analytics & Prediction Models

With Agile & DevOps becoming more & more predominant in software development methodologies, early performance analysis & predictive performance is becoming a norm for business critical & high traffic applications. Performance modelling & prediction analytics using the historical statistics gathered across several layers & that can help in analyzing several what-if scenarios & making quick performance judgement without actually testing the system.

So what is Performance Prediction?

It’s the process of modelling the system performance against simulated user loads by using mathematical expressions. Predictive models can only FORECAST, cannot ascertain what might happen in future as it is probabilistic in nature.

A Performance model uses specific number of building blocks to predict how performance will vary under different what-if scenarios like varied set of load conditions, change in workloads, and change in server capacity, etc.Usually the inputs to the model are expressed in mathematical quantities such as number of users, arrival rate, response time, throughput, resource utilization, etc.

How is Performance Analytics & Prediction correlated ?

A robust performance analytics solution should comprise of system performance data collected from production environment like server resource utilization monitoring, server health & performance metrics from APM tools, etc. A sophisticated analytics solution should have data collected at various stages of software development life cycle phases that can be correlated well using modelling & prediction techniques to forecast the system performance for great combination of what-if scenarios.

When the analytics solution build facilitating different types of data collection & storage used with the sound intelligence built through sound modelling & forecasting algorithms, it becomes predictive analytics which provides realistic forecasts on system performance for what-if scenarios.

There are 2 key types of models – Predictive & Prescriptive. A Predictive model built along with the intelligence to prescribe an action to business to act upon along with the feedback system that tracks outcome produced by the action taken becomes Prescriptive model. For example, a predictive model can predict the peak traffic throughput of the application under test whereas a prescriptive model can predict &recommend / alert business about the need to bring down the resident time of specific layer/method or to upgrade the specific hardware resource with high service demand to meet the performance SLAs with clear data points about the expected performance improvements.

Visualizing the required data as onion layers, the performance prediction accuracy increases when the data layers used for building the analytics solution increases. Some of the major data layers to be considered include performance modelling results & actual performance test results from controlled performance test environment, Network performance simulation results & device side performance metrics (for mobile applications), Test versus Production environment capacity differences , Production infrastructure monitoring statistics,website end user traffic patterns, & web (browser) performance statistics. At least to start with, production environment monitoring statistics & website user traffic statistics data layers are essential ones to do forecasts based on  historical data analysis using regression techniques.

                      

There are several open source & commercial tools generally used to perform the testing & analysis at each of the data layers represented in the figure. The key challenge lies in building the intelligence to parse the results produced by variety of tools & provision a tool agnostic generic reporting dashboard  that can be fed to the prediction / modelling algorithms to support in performance forecasting.

Details on modelling / prediction techniques will be available in a different post. 

Upsurge in Savvy Performance Analysis Tools leading to Enigmatic Trend

This article is also published in WOMEN TESTERS – APRIL 2016 EDITION
http://www.womentesters.com/2016/04/

Introduction

With the onset of SMAC (Social, Media, Analytics and Cloud) technologies& Digital transformations, there is an exponential increase in the level of knowledge expected from Performance Testers & Engineers. There has been a sudden increase in the number of vendors providing various application performance management tools to satisfy the growing demand on quick performance insights as high performance is one of the key factors driving the technological revolution. As Performance Engineering focuses on building systems with high performance and performing tuning & optimization to meet the performance, scalability & capacity demands, Performance Testers are usually seen inclined to have their career path towards Performance Engineering. 

This emerging trend in the rise of various performance testing &engineering toolsets for quick performance analysis &easy bottleneck detection has led to a confusing trend withinPerformance Testing/Engineering community.  The upcoming millennials seem to have developed more interest to get their hands on various tools rather than focusing on deepening their concepts &developing thought process for systematic performance problem diagnosis &tuning. This trend is creating a very big challenge for Performance COE’s (Center Of Excellence) as they strive towards successful project delivery in environments where these savvy tools don’t exists.

Tools are just Enablers; Don’t forget to add your Intelligence

The primary skills for Performance Testers is to design & execute various types of performance tests using variety of tools like HP Load Runner or JMeter. Performance Engineers through their vast performance testing experience for years or from developmental background have built stronger credibility as they understand the problems in application performance management including building scalable architecture solutions, selecting right design patterns that enhances performance, code profiling, etc. slightly better & know the areas to be analysed to identify the root cause of the problems when a performance SLA is not met.

Even now, a good performance tester is expected to have HP LoadRunner tool certification. There are no tool agnostic courses for acquiring performance testing capability. Due to which performance testers are usually seem biased & inclined towards specific tool terminologies & concepts. Majority of them think Performance testing is all about developing test scripts for the requested business use cases and create test scenario in the tool to execute load tests for the expected load conditions.

Performance Testers should firstly know how to strategize the tests, it’s not about just conducting load test or stress test using a tool.The key value adds in performance testing on how to bring in better accuracy & real time end user simulation in the performance tests, etc. Your favourite performance testing tool do not have the intelligence to report if the workload (use cases & load distribution) used for the test is incorrectly modelled. Many testers seem to be very comfortable with the knowledge of various tool specific settings without knowing the fact that a wrongly configured think time can end up running a completely wrong testnot even worth doing performance testing for your application.Majority of the production failure RCAs (root cause analysis) revealed improper & inaccurate performance tests due to incorrect design of the workload or improper NFRs used during validation tests.

The Performance testers should think beyond the performance testing tool & focus on gaining stronger understanding of the underlying Performance Testing & Engineering concepts to be able to run accurate & meaningful performance tests.

In recent past, there has been a sudden rise in the popularity of various commercial APM (Application Performance Monitoring) tools like Dynatrace, AppDynamics, New Relic, etc., while these tools have helped in quick & easy performance problem detection providing productivity boom due to lot of performance problem insights & readymade analysis recommendations in few button clicks, it is unfortunate that it has created a situation where Performance Testers /Engineers are more biased to gain experience on these tools rather than understanding the problem analysis concepts & principles.

The Performance Engineers we could find these days also have got biased to tool terminologies and tool provided metrics losing the big picture on overall application performance management & 360 degree of problem analysis. This lack of understanding in some cases have caused great embarrassments during project delivery. But I must agree the fact that these new generation tools are a great tribute for Performance Engineers & have brought our performance bottleneck analysis efforts from weeks to days thereby giving us room to fix it up quickly before it impacts the end user.

The bottom line fact is that there is no substitute for understanding of processes & Performance testing & Engineering concepts/principles which is required for managing performance throughout the SDLC life cycle phases. Gaining knowledge of the tools is important but not to be considered as a substitute for all the underlying performance testing/engineering principles. 

Tools are not an evil. They are great weapons to address the problem quickly. It’s just our perception to use them as enablers needs to be improved & we need to continue focusing on underlying Performance Testing/Engineering principles.

Use of RACI matrix to clarify your Performance Assessment Scope

Recently I had a performance assessment capability (& high level proposal) walkthrough discussion with one of our clients. I had several queries regarding the performance analysis activities scoped in for the engagement. Some of them are shared below:

• Who will perform the analysis on the ground?
• You will pinpoint there is a performance degradation issue & from there on is it our responsibility to analyze and fix it?
• My previous vendor helped me only to run the performance tests whereas its our sole responsibility to do any analysis/tuning to meet the performance SLAs ?
• You are quoting 45 person days effort for performance assessment whereas my current vendor did this using 1/3rd of your quoted efforts ?

I parked all those questions saying will revert back while doing walkthrough of the RACI Matrix slide.

When I moved on RACI Matrix slide, all I said is below :

Performance Testing scope of work will always have below characteristics

CLIENT SMEs are Accountable for ensuring application performance
PERFORMANCE TEAM is Responsible only for carrying out / execution of performance tests
CLIENT SMEs are Accountable & Consulted for performance bottleneck analysis and performance team supports by running retests to verify the performance issues are fixed.
PERFORMANCE TEAM is Informed once the performance fix ready for them to carry out retests.

whereas Performance Engineering scope will have the below characteristics

PERFORMANCE TEAM is Responsible for carrying out / execution of performance tests
PERFORMANCE TEAM is Accountable for ensuring the application performance. (Carryout all possible required analysis to meet the performance SLAs set by the business).
CLIENT SMEs are usually Informed & sometimes Consulted (on need basis for some key decision making) during the detailed performance bottleneck analysis.

Now, finally its your choice to demand what to expect from the performance excellence team.

I personally feel its our (Performance Excellence Team's) responsibility to bring awareness among all our stakeholders on these stuff.

Performance benchmarking assignments would have only subset of activities that is quoted as part of Performance testing assignment.

Performance testing assignment would have only a subset of activities that is quoted as part of performance engineering assignment.

As you move up in the ladder (Performance Benchmarking --> Performance Testing --> Performance Engineering), the scope of activities & effort requirement & thereby cost goes high.

Decide what is required based on the performance criticality of the application and budget availability.

Happy Testing!

Client-Side metrics versus Performance KPIs

Am happy to be back again blogging some useful stuff after a long time !!

Recently when i happened to have a discussion with one of the performance test engineer, I realized to bring this point elaborately looking at his misunderstanding.

The Client-Side performance is of great importance as it assesses end-user experience. It is vital to understand what is happening in the browser & its runtime (JavaScript , DOM). Optimizing the front-end performance from a single user point of view is very vital before testing an application with high user loads. The front-end, client-side performance optimization is quite simple compared to back-end performance optimization where lot of detailed analysis such as code profiling, hardware upgrade, etc. is required.

I experienced it several times & learnt in a hard way that client-side performance is always considered trivial as it is simple and can be quickly performed with right set of tools.

There are quite a good number of tools to do this analysis. To mention a few, Google's WebPageTest tool, Yahoo's YSlow extension, FireFox's FireBug extension, Google's pageSpeed, Ajax Dynatrace for IE, Speed Tracer extension for Google Chrome, etc.

The Key Performance Indicators (KPIs) like Response Time, Throughput, etc cannot be called as client-side metrics. May be you get to see these metrics through the performance test tool, but it is nothing to do with client-side metrics. The KPIs usually represent the system performance after the end of a performance test. These should be called as System Performance metrics or Performance KPIs.

Thanks for your Interest on my Performance Testing ebook

Dear Viewers,

Would like to send out my thanks message for all of you for all your appreciations for my book : Software Performance Testing Handbook - A Comprehensive guide for beginners. Am happy to share with you that the book has crossed 3000 downloads till now.

Thanks for your support and motivating me to do more.

Here is the free download link :

Software Performance Testing Handbook - A Comprehensive Guide for Begineers