When Fine-Tuning LLMs Meets Data Privacy: An Empirical Study of Federated Learning in LLM-Based Program Repair.
Abstract
When Fine-Tuning LLMs Meets Data Privacy: An Empirical Study of Federated Learning in LLM-Based Program Repair WENQIANG LUO, Department of Computer Science, City University of Hong Kong, China JACKY WAI KEUNG, Department of Computer Science, City University of Hong Kong, China BOYANG YANG, Jisuan Institute of Technology, Beijing JudaoYouda Network Technology Co. Ltd., China HE YE, School of Computer Science, Carnegie Mellon University, USA CLAIRE LE GOUES, School of Computer Science, Carnegie Mellon University, USA TEGAWENDÉ F. BISSYANDÉ, SnT, University of Luxembourg, Luxembourg HAOYE TIAN∗ , School of Computing and Information Systems, University of Melbourne, Australia BACH LE, School of Computing and Information Systems, University of Melbourne, Australia Software systems have been evolving rapidly and inevitably introducing bugs at an increasing rate, leading to significant losses in resources consumed by software maintenance. Recently, large language models (LLMs) have demonstrated remarkable potential in enhancing software development and maintenance practices, particularly in automated program repair (APR) with improved accuracy and efficiency of bug fixing. However, LLM-based APR heavily relies on high-quality code repositories. A larger portion of existing code repositories are for private use and proprietary assets from various industries, reflecting more diversity and nuances in the data since real-world industries often have more extensive software development practices, which cannot be covered by merely public datasets. Therefore, utilizing private datasets shows significant potential in enhancing software development and maintenance. However, obtaining such data from various industries is hindered by data privacy concerns, as companies are reluctant to share their proprietary codebases. Nevertheless, there has also been no in-depth investigation of collaborative software development by learning from private and decentralized data while preserving data privacy for program repair in prior works. To address the gap, we investigate the use of federated learning as a privacy-preserving approach that enables private entities to fine-tune LLMs on proprie