title: Bytes Are All You Need: Transformers Operating Directly On File Bytes
summary: Modern deep learning approaches usually transform inputs into a
  modality-specific form. For example, the most common deep learning approach to
  image classification involves decoding image file bytes into an RGB tensor
  which is passed into a neural network. Instead, we investigate performing
  classification directly on file bytes, without the need for decoding files at
  inference time. Using file bytes as model inputs enables the development of
  models which can operate on multiple input modalities. Our model,
  ByteFormer, achieves an ImageNet Top-1 classification accuracy of
  77.33% when training and testing directly on TIFF file bytes using a
  transformer backbone with configuration similar to DeiT-Ti (72.2% accuracy
  when operating on RGB images). Without modifications or hyperparameter tuning,
  ByteFormer achieves 95.42% classification accuracy when operating on WAV
  files from the Speech Commands v2 dataset (compared to state-of-the-art
  accuracy of 98.7%). Additionally, we demonstrate that ByteFormer has
  applications in privacy-preserving inference. ByteFormer is capable of
  performing inference on particular obfuscated input representations with no
  loss of accuracy. We also demonstrate ByteFormer's ability to perform inference
  with a hypothetical privacy-preserving camera which avoids forming full images
  by consistently masking 90% of pixel channels, while still achieving
  71.35% accuracy on ImageNet. Our code will be made available at
  https://github.com/apple/ml-cvnets/tree/main/examples/byteformer.
paper_authors._id: 647977a8ae0fa9dc233837bc
paper_authors.name: Maxwell Horton
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-05T10:07:54.611Z
paper_authors._id: 647a5afc732a0cd866eb512f
paper_authors.avatarUrl: /avatars/fbd0ba2a7ba70414ac903e583ce62933.svg
paper_authors.fullname: Maxwell Horton
paper_authors.type: user
paper_authors.user: mchorton
paper_authors._id: 647977a8ae0fa9dc233837bd
paper_authors.name: Sachin Mehta
paper_authors._id: 647977a8ae0fa9dc233837be
paper_authors.name: Ali Farhadi
paper_authors._id: 647977a8ae0fa9dc233837bf
paper_authors.name: Mohammad Rastegari
paper_title: Bytes Are All You Need: Transformers Operating Directly On File Bytes
paper_summary: Modern deep learning approaches usually transform inputs into a
  modality-specific form. For example, the most common deep learning approach to
  image classification involves decoding image file bytes into an RGB tensor
  which is passed into a neural network. Instead, we investigate performing
  classification directly on file bytes, without the need for decoding files at
  inference time. Using file bytes as model inputs enables the development of
  models which can operate on multiple input modalities. Our model,
  ByteFormer, achieves an ImageNet Top-1 classification accuracy of
  77.33% when training and testing directly on TIFF file bytes using a
  transformer backbone with configuration similar to DeiT-Ti (72.2% accuracy
  when operating on RGB images). Without modifications or hyperparameter tuning,
  ByteFormer achieves 95.42% classification accuracy when operating on WAV
  files from the Speech Commands v2 dataset (compared to state-of-the-art
  accuracy of 98.7%). Additionally, we demonstrate that ByteFormer has
  applications in privacy-preserving inference. ByteFormer is capable of
  performing inference on particular obfuscated input representations with no
  loss of accuracy. We also demonstrate ByteFormer's ability to perform inference
  with a hypothetical privacy-preserving camera which avoids forming full images
  by consistently masking 90% of pixel channels, while still achieving
  71.35% accuracy on ImageNet. Our code will be made available at
  https://github.com/apple/ml-cvnets/tree/main/examples/byteformer.
paper_ai_keywords: ByteFormer ImageNet TIFF file bytes transformer backbone DeiT-Ti WAV files Speech Commands v2 privacy-preserving inference obfuscated input representations privacy-preserving camera pixel channels