Enhancing Valid Test Input Generation with Distribution Awareness for Deep Neural Networks.
Abstract
Comprehensive testing is important in improving the reliability of Deep Learning (DL)-based systems. Various Test Input Generators (TIGs) have been proposed to generate misbehavior-inducing test inputs. However, the lack of validity checking in TIGs often results in the generation of invalid inputs (i.e., out of the learned distribution), leading to unreliable testing. To save the effort of manually checking the validity and improve test efficiency, it is important to assess the effectiveness and reliability of automated validators. In this study, we comprehensively assess four automated Input Validators (IVs). Our findings show that the accuracy of IVs ranges from 49% to 77%. Distance-based IVs generally outperform reconstruction-based and density-based IVs for both classification and regression tasks. Based on the findings, we enhance existing testing frameworks by incorporating distribution awareness through joint optimization. The results demonstrate our framework leads to a 2% to 10% increase in the number of valid inputs, which establishes our method as an effective technique for valid test input generation. Index Terms-Input Validation, Anomaly Detection, Deep Learning, Software Testing I. I NTRODUCTION With the rapid development of deep learning (DL), many software applications have integrated deep neural networks (DNNs) to improve their performance. Despite the notable achievements of DL-based models across various domains [1]-[3], research has indicated their vulnerability to malicious attacks, necessitating comprehensive testing [4]-[7]. Thorough testing of a DL system requires diverse and valid test inputs which can trigger model misbehavior. While plenty of Test Input Generators (TIGs) [4]-[6], [8]-[11] have been proposed to efficiently generate error-inducing inputs, most of them fall short in checking the validity of the generated test inputs. Input validation is a fundamental concept in software testing. In traditional software engineering, developers must create valid test cases to effectively evaluate the functional logic of the software and ensure compliance with specifications. When an invalid test case is encountered, the software will handle th