Journey towards Performance Analytics & Prediction Models

With Agile & DevOps becoming more & more predominant in software development methodologies, early performance analysis & predictive performance is becoming a norm for business critical & high traffic applications. Performance modelling & prediction analytics using the historical statistics gathered across several layers & that can help in analyzing several what-if scenarios & making quick performance judgement without actually testing the system.

So what is Performance Prediction?

It’s the process of modelling the system performance against simulated user loads by using mathematical expressions. Predictive models can only FORECAST, cannot ascertain what might happen in future as it is probabilistic in nature.

A Performance model uses specific number of building blocks to predict how performance will vary under different what-if scenarios like varied set of load conditions, change in workloads, and change in server capacity, etc.Usually the inputs to the model are expressed in mathematical quantities such as number of users, arrival rate, response time, throughput, resource utilization, etc.

How is Performance Analytics & Prediction correlated ?

A robust performance analytics solution should comprise of system performance data collected from production environment like server resource utilization monitoring, server health & performance metrics from APM tools, etc. A sophisticated analytics solution should have data collected at various stages of software development life cycle phases that can be correlated well using modelling & prediction techniques to forecast the system performance for great combination of what-if scenarios.

When the analytics solution build facilitating different types of data collection & storage used with the sound intelligence built through sound modelling & forecasting algorithms, it becomes predictive analytics which provides realistic forecasts on system performance for what-if scenarios.

There are 2 key types of models – Predictive & Prescriptive. A Predictive model built along with the intelligence to prescribe an action to business to act upon along with the feedback system that tracks outcome produced by the action taken becomes Prescriptive model. For example, a predictive model can predict the peak traffic throughput of the application under test whereas a prescriptive model can predict &recommend / alert business about the need to bring down the resident time of specific layer/method or to upgrade the specific hardware resource with high service demand to meet the performance SLAs with clear data points about the expected performance improvements.

Visualizing the required data as onion layers, the performance prediction accuracy increases when the data layers used for building the analytics solution increases. Some of the major data layers to be considered include performance modelling results & actual performance test results from controlled performance test environment, Network performance simulation results & device side performance metrics (for mobile applications), Test versus Production environment capacity differences , Production infrastructure monitoring statistics,website end user traffic patterns, & web (browser) performance statistics. At least to start with, production environment monitoring statistics & website user traffic statistics data layers are essential ones to do forecasts based on  historical data analysis using regression techniques.

                      

There are several open source & commercial tools generally used to perform the testing & analysis at each of the data layers represented in the figure. The key challenge lies in building the intelligence to parse the results produced by variety of tools & provision a tool agnostic generic reporting dashboard  that can be fed to the prediction / modelling algorithms to support in performance forecasting.

Details on modelling / prediction techniques will be available in a different post.