Towards Understanding Bugs in Distributed Training and Inference Frameworks for Large Language Models.
Abstract
With the rapid advancement of large language models (LLMs), distributed training and inference frameworks such as DeepSpeed have become essential infrastructure for scaling both model training and inference across multiple GPUs or nodes. However, the growing complexity of these frameworks introduces non-trivial software bugs that can impair training performance, cause unexpected failures, and lead to significant resource waste. Understanding the characteristics of framework bugs is a foundational step toward quality assurance, enabling the design of more effective debugging and repair approaches. Therefore, our paper presents the first large-scale empirical study of 308 resolved bugs from three widely used distributed training and inference frameworks: DeepSpeed, Megatron-LM, and Colossal-AI. We examine bug symptoms, root causes, bug identification and fixing efforts, and common low-effort fixing strategies. Our findings show that the most frequent symptoms include crash, incorrect functionality, and build failure, while the bug root causes are primarily API misuse, incorrect implementation, configuration error, and missing preconditions. Additionally, the distributed nature of these frameworks introduces unique bug root causes, such as allocation strategy error and distributed communication error. Diagnosing and fixing complex bugs remains challenging due to factors like the disconnect between symptoms and root causes, high bug reproduction costs, and low-level or crosscomponent interactions. Interestingly, we observe that 48% of bug fixes require minimal code changes (≤ 10 LOC) and follow simple strategies such as conditional logic optimization, parameter handling enhancement, or version compatibility handling, indicating potential for automation. Based on these insights, we offer several implications for improving the reliability of both distributed training and inference frameworks and their dependent LLM projects, while also identifying opportunities to leverage LLM-based tools for automated debugging and repair. I. I NTRODUCTION In recent years, the field of artificial intelligence has witnessed exponential growth in model size, architectural complexity