A Comprehensive Study of Bugs in Modern Distributed Deep Learning Systems.
Abstract
A Comprehensive Study of Bugs in Modern Distributed Deep Learning Systems XIAOXUE MA, Hong Kong Metropolitan University, China WANWEI ZHAN, Wuhan University of Technology, China JIALE CHEN, Wuhan University of Technology, China YISHU LI, Hong Kong Metropolitan University, China JACKY KEUNG, City University of Hong Kong, China FEDERICA SARRO, University College London, UK In today's data-driven era, deep learning is vital for processing massive datasets, yet single-device training is constrained by computational and memory limits. Distributed deep learning overcomes these challenges by leveraging multiple GPUs or machines in parallel. While general-purpose frameworks (e.g., TensorFlow and PyTorch) provide distributed capabilities, these are often add-on features that demand significant manual effort for advanced parallelism, underscoring the need for specialized frameworks. This study conducts the first large-scale empirical analysis of practitioner challenges in dedicated distributed frameworks. We examine 849 real-world issues from DeepSpeed, Megatron-LM, and Colossal-AI and construct a taxonomy of 34 bug symptoms, 28 root causes, and 6 fix patterns. Crucially, we establish explicit mappings between symptoms, causes, and fixes across distributed training stages, enabling a systematic understanding of how issues emerge and are resolved. Our results show that 45.1% of bug symptoms are unique to distributed frameworks, with setup failures, memory issues, and performance anomalies being the most prevalent. Moreover, 95% of issues in the communication setup stage occur exclusively in distributed contexts. We also find over 60% of cases can be resolved through version and dependency management, and distributed feature, API, and communication tuning. Based on these findings, we provide actionable implications. CCS Concepts: • Software and its engineering → Empirical software validation. Additional Key Words and Phrases: Empirical study, bug-fix pattern, distributed deep learning ACM Reference Format: Xiaoxue Ma, Wanwei Zhan, Jiale Chen, Yishu Li, Jacky Keung, and Federica Sarro. 2018. A Comprehensive Study of Bugs in Modern Distributed Deep Learning Systems. J. ACM 37