title: Interface Design for Self-Supervised Speech Models
summary: Self-supervised speech (SSL) models have recently become widely adopted for
  many downstream speech processing tasks. The general usage pattern is to employ
  SSL models as feature extractors, and then train a downstream prediction head
  to solve a specific task. However, different layers of SSL models have been
  shown to capture different types of information, and the methods of combining
  them are not well studied. To this end, we extend the general framework for SSL
  model utilization by proposing the interface that connects the upstream and
  downstream. Under this view, the dominant technique of combining features via a
  layerwise weighted sum can be regarded as a specific interface. We propose
  several alternative interface designs and demonstrate that the weighted sum
  interface is suboptimal for many tasks. In particular, we show that a
  convolutional interface whose depth scales logarithmically with the depth of
  the upstream model consistently outperforms many other interface designs.
paper_authors._id: 66742e344dd90a98c21db5ba
paper_authors.name: Yi-Jen Shih
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-06-20T13:27:16.850Z
paper_authors._id: 62de905d35a8bfe88fc091a5
paper_authors.avatarUrl: /avatars/b9d289319f44231b6e50de491f0b3c30.svg
paper_authors.fullname: Ian Shih
paper_authors.type: user
paper_authors.user: atosystem
paper_authors._id: 66742e344dd90a98c21db5bb
paper_authors.name: David Harwath
paper_title: Interface Design for Self-Supervised Speech Models
paper_summary: Self-supervised speech (SSL) models have recently become widely adopted for
  many downstream speech processing tasks. The general usage pattern is to employ
  SSL models as feature extractors, and then train a downstream prediction head
  to solve a specific task. However, different layers of SSL models have been
  shown to capture different types of information, and the methods of combining
  them are not well studied. To this end, we extend the general framework for SSL
  model utilization by proposing the interface that connects the upstream and
  downstream. Under this view, the dominant technique of combining features via a
  layerwise weighted sum can be regarded as a specific interface. We propose
  several alternative interface designs and demonstrate that the weighted sum
  interface is suboptimal for many tasks. In particular, we show that a
  convolutional interface whose depth scales logarithmically with the depth of
  the upstream model consistently outperforms many other interface designs.
paper_ai_keywords: self-supervised speech (SSL) models feature extractors prediction head layerwise weighted sum convolutional interface logarithmically scaled depth