Progressive Token Length Scaling in Transformer Encoders for Efficient Universal Segmentation
<|reference_start|>Progressive Token Length Scaling in Transformer Encoders for Efficient Universal Segmentation: A powerful architecture for universal segmentation relies on transformers that encode multi-scale image features and decode object queries into mask predictions. With efficiency being a high priority for scaling such models, we observed that the state-of-the-art method Mask2Former uses ~50% of its compute only on the transformer encoder. This is due to the retention of a full-length token-level representation of all backbone feature scales at each encoder layer. With this observation, we propose a strategy termed PROgressive Token Length SCALing for Efficient transformer encoders (PRO-SCALE) that can be plugged-in to the Mask2Former-style segmentation architectures to significantly reduce the computational cost. The underlying principle of PRO-SCALE is: progressively scale the length of the tokens with the layers of the encoder. This allows PRO-SCALE to reduce computations by a large margin with minimal sacrifice in performance (~52% GFLOPs reduction with no drop in performance on COCO dataset). We validate our framework on multiple public benchmarks.<|reference_end|>
@article{aich2024progressive,
    title={Progressive Token Length Scaling in Transformer Encoders for Efficient
  Universal Segmentation},
    author={Abhishek Aich, Yumin Suh, Samuel Schulter, Manmohan Chandraker},
    journal={arXiv preprint arXiv:2404.14657},
    year={2024},
    archivePrefix={arXiv},
    eprint={2404.14657},
    primaryClass={cs.CV}
}