Beyond Log Parsers: A Scalable AI-Driven Framework for Efficient Log Anomaly Detection in Software Engineering.
Abstract
Log anomaly detection is critical for ensuring software system reliability and security, yet challenges persist in log parser dependency, small-scale dataset applicability, and hyperparameter tuning efficiency. Existing methods over-rely on predefined log templates, leading to information loss and high computational overhead. Additionally, anomaly detection models often struggle with limited log data, and hyperparameter tuning remains computationally expensive in dynamic environments. In this paper, we empirically evaluate seven state-of-the-art anomaly detection models across varied software systems, assessing the necessity of log parsers and model performance on small-scale datasets. Furthermore, we propose SMAC-<T>, an enhanced real-time hyperparameter optimization framework, integrating stochastic gradient descent (SGD) and adaptive learning to improve model adaptability and efficiency. Our experiments on six benchmark datasets demonstrate that SMAC-<T> achieves an overall average F1-score improvement of 4.27%, a 27.55% reduction in hyperparameter tuning time compared to other models, and a 1.35% increase in F1-score when adapting to newly emerging logs, compared to its counterpart without SGD integration. These findings underscore the practical advantages of AI-driven log analysis, providing valuable insights into scalable, software-engineered anomaly detection. Index Terms-AI-Driven Log Anomaly Detection, Real-Time Hyperparameter Optimization, Log Analysis, Scalable Log Parsing and Detection, Empirical Software Anomaly Analysis I. I NTRODUCTION Log anomaly detection plays a vital role in modern software engineering by enabling proactive failure prevention [1]. As software systems scale, the volume and heterogeneity of logs increase, rendering manual inspection impractical and making automated detection essential [2]. However, traditional approaches heavily depend on log parsers, which introduce significant computational overhead, enforce rigid template dependencies, and risk discarding anomaly-related information [3], [4]. Studies show that most log parsers cannot process largescale datasets efficiently [5], and even state-of-the-art models like ChatGPT