title: Training Noise Token Pruning
summary: In the present work we present Training Noise Token (TNT) Pruning for vision
  transformers. Our method relaxes the discrete token dropping condition to
  continuous additive noise, providing smooth optimization in training, while
  retaining discrete dropping computational gains in deployment settings. We
  provide theoretical connections to Rate-Distortion literature, and empirical
  evaluations on the ImageNet dataset using ViT and DeiT architectures
  demonstrating TNT's advantages over previous pruning methods.
paper_authors._id: 674e25943cf5e90cbc7f0e9e
paper_authors.name: Mingxing Rao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-03T11:16:04.218Z
paper_authors._id: 6648ce9df16e33cc991f2b94
paper_authors.avatarUrl: /avatars/a3e4c39146b34ec82ab0eb31444b678c.svg
paper_authors.fullname: Mingxing Rao
paper_authors.type: user
paper_authors.user: ethanrao1
paper_authors._id: 674e25943cf5e90cbc7f0e9f
paper_authors.name: Bohan Jiang
paper_authors._id: 674e25943cf5e90cbc7f0ea0
paper_authors.name: Daniel Moyer
paper_title: Training Noise Token Pruning
paper_summary: In the present work we present Training Noise Token (TNT) Pruning for vision
  transformers. Our method relaxes the discrete token dropping condition to
  continuous additive noise, providing smooth optimization in training, while
  retaining discrete dropping computational gains in deployment settings. We
  provide theoretical connections to Rate-Distortion literature, and empirical
  evaluations on the ImageNet dataset using ViT and DeiT architectures
  demonstrating TNT's advantages over previous pruning methods.
paper_ai_keywords: Training Noise Token (TNT) Pruning vision transformers token dropping continuous additive noise smooth optimization Rate-Distortion literature ImageNet dataset ViT (Vision Transformer) architectures DeiT (Distilled Vision Transformer) architectures