AdaPTwin: Low-Cost Adaptive Compression of Product Twins in Transformers
<|reference_start|>AdaPTwin: Low-Cost Adaptive Compression of Product Twins in Transformers: While large transformer-based models have exhibited remarkable performance in speaker-independent speech recognition, their large size and computational requirements make them expensive or impractical to use in resource-constrained settings. In this work, we propose a low-rank adaptive compression technique called AdaPTwin that jointly compresses product-dependent pairs of weight matrices in the transformer attention layer. Our approach can prioritize the compressed model's performance on a specific speaker while maintaining generalizability to new speakers and acoustic conditions. Notably, our technique requires only 8 hours of speech data for fine-tuning, which can be accomplished in under 20 minutes, making it highly cost-effective compared to other compression methods. We demonstrate the efficacy of our approach by compressing the Whisper and Distil-Whisper models by up to 45% while incurring less than a 2% increase in word error rate.<|reference_end|>
@article{biju2024adaptwin:,
    title={AdaPTwin: Low-Cost Adaptive Compression of Product Twins in Transformers},
    author={Emil Biju, Anirudh Sriram, Mert Pilanci},
    journal={arXiv preprint arXiv:2406.08904},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.08904},
    primaryClass={cs.LG cs.SD eess.AS}
}