Cross-Modal and Cross-Domain Knowledge Transfer for Label-Free 3D Segmentation.
Abstract
. Current state-of-the-art point cloud-based perception methods usually rely on large-scale labeled data, which requires expensive manual annotations. A natural option is to explore the unsupervised methodology for 3D perception tasks. However, such methods often face substantial performance-drop difficulties. Fortunately, we found that there exist amounts of image-based datasets and an alternative can be proposed, i.e., transferring the knowledge in the 2D images to 3D point clouds. Specifically, we propose a novel approach for the challenging cross-modal and cross-domain adaptation task by fully exploring the relationship between images and point clouds and designing effective feature alignment strategies. Without any 3D labels, our method achieves state-of-the-art performance for 3D point cloud semantic segmentation on SemanticKITTI by using the knowledge of KITTI360 and GTA5, compared to existing unsupervised and weakly-supervised baselines. Keywords: Point Cloud Semantic Segmentation · Unsupervised Domain Adaptation · Cross-modal Transfer Learning 1 Introduction Accurate depth information of large-scale scenes captured by LiDAR is the key to 3D perception. Consequently, LiDAR point cloud-based research has gained popularity. However, existing learning-based methods [10,11,37] demand extensive training data and significant labor power. Especially for the 3D segmentation task, people need to assign class labels to massive amounts of points. Such high cost really hinders the progress of related research. Therefore, relieving the annotation burden has become increasingly important for 3D perception. J. Zhang and H. Yang - Equal contribution. c The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2024 ! Q. Liu et al. (Eds.): PRCV 2023, LNCS 14427, pp. 465-477, 2024. https://doi.org/10.1007/978-981-99-8435-0_37 466 J. Zhang et al. Recently, plenty of label-efficient methods have been proposed for semantic segmentation. They try to simplify the labeling operation by using scene-level or sub-cloud-level weak labels [24], or utilizing clicks or line segments as objectlevel or class-level weak labels [19,20]. However, these methods stil