When Fine-Tuning LLMs Meets Data Privacy: An Empirical Study of Federated Learning in LLM-Based Program Repair.
Abstract
When Fine-Tuning LLMs Meets Data Privacy: An Empirical Study of Federated Learning in LLM-Based Program Repair WENQIANG LUO and JACKY WAI KEUNG, Department of Computer Science, City University of Hong Kong, Hong Kong, Hong Kong BOYANG YANG, Jisuan Institute of Technology, Beijing JudaoYouda Network Technology Co. Ltd., Beijing, China HE YE and CLAIRE LE GOUES, School of Computer Science, Carnegie Mellon University, Pittsburgh, Pennsylvania, USA TEGAWENDÉ F. BISSYANDÉ, SnT, University of Luxembourg, Esch-sur-Alzette, Luxembourg HAOYE TIAN, School of Computing and Information Systems, University of Melbourne, Melbourne, Australia BACH LE, School of Computing and Information Systems, The University of Melbourne, Melbourne, Australia Software systems have been evolving rapidly and inevitably introducing bugs at an increasing rate, leading to significant maintenance costs. While large language models (LLMs) have demonstrated remarkable potential in enhancing software development and maintenance practices, particularly in automated program repair (APR), they rely heavily on high-quality code repositories. Most code repositories are proprietary assets that capture the diversity and nuances of real-world industry software practices, which public datasets cannot fully represent. However, obtaining such data from various industries is hindered by data privacy concerns, as companies are reluctant to share their proprietary codebases. There has also been no in-depth investigation of collaborative software development by learning from private and decentralized data while preserving data privacy for program repair. To address the gap, we investigate federated learning as a privacy-preserving method for fine-tuning LLMs on proprietary and decentralized data to boost collaborative software development and maintenance. We use the private industrial dataset TutorCode for fine-tuning and the EvalRepair-Java benchmark for evaluation, and assess whether federated fine-tuning enhances program repair. We then further explore how code heterogeneity This research was funded (partially) by the Australian Government through the Australian Research Council's Discovery Early Career Research