A Semisupervised Approach for Industrial Anomaly Detection via Self-Adaptive Clustering.
Abstract
With the rapid development of the Industrial Internet of Things, log-based anomaly detection has become vital for smart industrial construction that has prompted many researchers to contribute. To detect anomalies based on log data, semisupervised approaches stand out from supervised and unsupervised approaches because they only require a portion of labeled data and are relatively stable. However, the state-of-the-art semisupervised approaches still suffer from two main problems: manual parameter setting and unsatisfactory performance with high false positives. We propose AdaLog, an integrated semisupervised approach based on self-adaptive clustering, for industrial anomaly detection. In particular, the clustering step performs automatic label probability estimation by distinguishing 12 situations so that the label probability of each unlabeled data can be carefully calculated, leading to high accuracy. In addition, AdaLog employs a pretrained model to learn contextual information comprehensively and a transformer-based model to detect anomalies efficiently. To alleviate class imbalance, an undersampling method is incorporated. The results on three popular datasets demonstrate that AdaLog significantly outperforms three stateof-the-art semisupervised approaches by 17.8%-2489.8% on average in terms of F1-score, and is even superior to two supervised approaches in most cases with average improvements of 10.9%-23.8%. Index Terms-Clustering, deep learning, intelligent anomaly detection, transformer. Manuscript received 21 December 2022; revised 28 March 2023; accepted 7 May 2023. Date of publication 29 May 2023; date of current version 19 January 2024. This work was supported in part by the General Research Fund of the Research Grants Council of Hong Kong under Grant 11208017; in part by the research funds of the City University of Hong Kong under Grant 7005028, Grant 7005217, and Grant 6000796; and in part by other industry projects under Grant 9229109, Grant 9229098, Grant 9229029, Grant 9220103, Grant 9220097, and Grant 9440227. Paper no. TII-22-5179. (Corresponding authors: Yan Xiao; Xiao Yu.) Xiaoxue Ma, Jacky Keung, and Yishu Li are with the Department of Co