Hybrid Privacy Policy-Code Consistency Check using Knowledge Graphs and LLMs.
Abstract
The increasing concern in user privacy misuse has accelerated research into checking consistencies between smartphone apps' declared privacy policies and their actual behaviors. Recent advances in Large Language Models (LLMs) have introduced promising techniques for semantic comparison, but these methods often suffer from low accuracies and expensive computational costs. To address this problem, this paper proposes a novel hybrid approach that integrates 1) knowledge graph-based deterministic checking to ensure higher accuracy, and 2) LLMs exclusively used for preliminary semantic analysis to save computational costs. Preliminary evaluation indicates this hybrid approach not only achieves 37.63% increase in precision and 23.13% increase F1-score but also consumes 93.5% less tokens and 87.3% shorter time. 2. P ROPOSED H YBRID A PPROACH Figure 1 presents an overview of the hybrid approach, realized through coordinated interaction among three core components: a Policy Reader, a Leak Extractor, and a Consistency Checker. Keywords-Privacy Alignment, Privacy Testing, Large Language Models, Knowledge Graph, Static Analysis 1. I NTRODUCTION Smartphone apps have transformed modern life, influencing nearly every aspect of human activity ranging from daily purchases to civic engagement. These apps typically justify their extensive data collection practices as necessary for service improvement, however, recently there is a rising concern in the misuse of these data. While developers are required to disclose their data practices through privacy policies, numerous studies have highlighted significant gaps between these formal declarations and the apps' actual data handling behaviors [1]. Privacy policy-code consistency check has emerged as a significant research stream [2], [3], where most studies employ static analysis tools like FlowDroid for this task [1], [4]. Recent work has explored LLM-based approaches for semantic comparison [5], typically implementing a pipeline that: 1) extracts events and data flows, 2) identifies relevant methods, and 3) computes policy-code similarity. However, the exclusive LLMs-based approach suffers from two fundamental limitations: 1) Over