Quantitative Analysis for Non-linear System Performance Data Using Case-Based Reasoning.
Abstract
Effective software architecture evaluation methods are essential in today's system development for mission critical systems. We have previously developed MEMS and a set of test statistics for evaluating middleware architectures, which proven an effective assessment of important quality attributes and their characterizations. We have observed it is common that many system performance response data are not of linear nature, where using linear modeling is not feasible in these scenarios for system performance predictions. To provide an alternative quantitative assessment on the system performance using actual runtime datasets, we developed a set of non-linear analysis procedure based on Case-based Reasoning (CBR), a machine learning method widely used in another disciplines of Software Engineering. Experiments were carried out based on actual runtime performance datasets. Results confirm that our non-linear analysis method CBR4MEMS produced accurate performance predictions and outperformed linear approaches. Our approach utilizing CBR to enable performance assessments on non-linear datasets, a major step forward to support software architecture evaluation. system performance experimental data derived from the middleware architecture evaluation test bed environment for airborne mission critical systems. Linear modeling methods have been used in the previous studies[1, 2], result shows that many system performance datasets under investigation were not of linear nature, therefore alternative non-linear methods are required to fully investigate the nature of the data. This paper introduces a novel non-linear modeling approach using Case-based Reasoning (CBR) from the artificial intelligence research field. A tailored approach for middleware architecture evaluation combining a sensitivity procedure has been developed in the study, namely CBR4MEMS. Software systems may response differently due to different characteristics in the middleware architectural configurations and the execution scenarios in the experiments. Therefore systems will behave differently but not necessarily in a linear and predictable fashion. CBR4MEMS utilizes K-NN (k-nearest neighbor) algorithm to f