TTT-KD: Test-Time Training for 3D Semantic Segmentation through Knowledge Distillation from Foundation Models
<|reference_start|>TTT-KD: Test-Time Training for 3D Semantic Segmentation through Knowledge Distillation from Foundation Models: Test-Time Training (TTT) proposes to adapt a pre-trained network to changing data distributions on-the-fly. In this work, we propose the first TTT method for 3D semantic segmentation, TTT-KD, which models Knowledge Distillation (KD) from foundation models (e.g. DINOv2) as a self-supervised objective for adaptation to distribution shifts at test-time. Given access to paired image-pointcloud (2D-3D) data, we first optimize a 3D segmentation backbone for the main task of semantic segmentation using the pointclouds and the task of 2D $\to$ 3D KD by using an off-the-shelf 2D pre-trained foundation model. At test-time, our TTT-KD updates the 3D segmentation backbone for each test sample, by using the self-supervised task of knowledge distillation, before performing the final prediction. Extensive evaluations on multiple indoor and outdoor 3D segmentation benchmarks show the utility of TTT-KD, as it improves performance for both in-distribution (ID) and out-of-distribution (ODO) test datasets. We achieve a gain of up to 13% mIoU (7% on average) when the train and test distributions are similar and up to 45% (20% on average) when adapting to OOD test samples.<|reference_end|>
@article{weijler2024ttt-kd:,
    title={TTT-KD: Test-Time Training for 3D Semantic Segmentation through
  Knowledge Distillation from Foundation Models},
    author={Lisa Weijler and Muhammad Jehanzeb Mirza and Leon Sick and Can Ekkazan
  and Pedro Hermosilla},
    journal={arXiv preprint arXiv:2403.11691},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.11691},
    primaryClass={cs.CV}
}