title: Large Language Models Implicitly Learn to See and Hear Just By Reading
summary: This paper presents a fascinating find: By training an auto-regressive LLM
  model on text tokens, the text model inherently develops internally an ability
  to understand images and audio, thereby developing the ability to see and hear
  just by reading. Popular audio and visual LLM models fine-tune text LLM models
  to give text output conditioned on images and audio embeddings. On the other
  hand, our architecture takes in patches of images, audio waveforms or tokens as
  input. It gives us the embeddings or category labels typical of a
  classification pipeline. We show the generality of text weights in aiding audio
  classification for datasets FSD-50K and GTZAN. Further, we show this working
  for image classification on CIFAR-10 and Fashion-MNIST, as well on image
  patches. This pushes the notion of text-LLMs learning powerful internal
  circuits that can be utilized by activating necessary connections for various
  applications rather than training models from scratch every single time.
paper_authors._id: 6833cb25fe87d9433dfd2b1c
paper_authors.name: Prateek Verma
paper_authors._id: 6833cb25fe87d9433dfd2b1d
paper_authors.name: Mert Pilanci
paper_title: Large Language Models Implicitly Learn to See and Hear Just By Reading
paper_summary: This paper presents a fascinating find: By training an auto-regressive LLM
  model on text tokens, the text model inherently develops internally an ability
  to understand images and audio, thereby developing the ability to see and hear
  just by reading. Popular audio and visual LLM models fine-tune text LLM models
  to give text output conditioned on images and audio embeddings. On the other
  hand, our architecture takes in patches of images, audio waveforms or tokens as
  input. It gives us the embeddings or category labels typical of a
  classification pipeline. We show the generality of text weights in aiding audio
  classification for datasets FSD-50K and GTZAN. Further, we show this working
  for image classification on CIFAR-10 and Fashion-MNIST, as well on image
  patches. This pushes the notion of text-LLMs learning powerful internal
  circuits that can be utilized by activating necessary connections for various
  applications rather than training models from scratch every single time.
paper_ai_summary: Auto-regressive text LLMs trained on text can develop internal capabilities for understanding images and audio, enabling them to perform classification tasks across different modalities without fine-tuning.
paper_ai_keywords: auto-regressive LLM text tokens audio visual embeddings category labels classification FSD-50K GTZAN CIFAR-10 Fashion-MNIST image patches