A Novel Semi-Supervised Model for Generalizing Log Anomaly Detection with Limited Labeled Data.
Abstract
Log anomaly detection is critical for ensuring the reliability and stability of large-scale distributed systems by identifying unusual patterns in system logs. However, existing methods often fail to generalize across diverse datasets, hindering their scalability and robustness. This paper proposes a novel anomaly detection model that integrates an LLMbased log parser, Generative Adversarial Networks (GANs) for anomaly detection, and Permutation Event Modeling (PEM) to capture temporal dependencies in log events, using only 5% of labeled logs for training. Our model leverages GANs to generate synthetic log data, enabling it to generalize effectively across diverse datasets with varying system architectures and log formats. By reducing the reliance on labeled data and enhancing generalization, our approach improves performance and scalability, offering a robust solution for log anomaly detection in dynamic environments. across varying system architectures, log formats, and operational conditions, has not been thoroughly explored in existing research. To address these challenges, we propose the use of GANs for anomaly detection, which is well-suited for modeling the distribution of normal log events. In GANs, the generator learns to produce synthetic log sequences that resemble normal data, while the discriminator detects deviations from this normal behavior, identifying anomalies. This approach is particularly effective because GANs can learn from very limited data, making them a powerful tool in scenarios where labeled data is scarce. However, applying GANs to log anomaly detection presents several challenges, including the need for high-quality, contextually rich log data to ensure accurate model training. To address this, we use an LLM-based parser for log parsing, ensuring high accuracy in event extraction. Additionally, GANs face difficulties in capturing the sequential and temporal dependencies inherent in log data, which are crucial for detecting complex anomalies. To overcome this, we employ 5% of labeled logs to build a diversified training set, reducing the inefficiency of manual labeling, and introduce Permutation Event Modeling (PEM), a method that