OWSM v3.1: Better and Faster Open Whisper-Style Speech Models based on
  E-Branchformer
Recent studies have advocated for fully open foundation models to promote
transparency and open science. As an initial step, the Open Whisper-style
Speech Model (OWSM) reproduced OpenAI's Whisper using publicly available data
and open-source toolkits. With the aim of reproducing Whisper, the previous
OWSM v1 through v3 models were still based on Transformer, which might lead to
inferior performance compared to other state-of-the-art speech encoders. In
this work, we aim to improve the performance and efficiency of OWSM without
extra training data. We present E-Branchformer based OWSM v3.1 models at two
scales, i.e., 100M and 1B. The 1B model is the largest E-Branchformer based
speech model that has been made publicly available. It outperforms the previous
OWSM v3 in a vast majority of evaluation benchmarks, while demonstrating up to
25% faster inference speed. We publicly release the data preparation scripts,
pre-trained models and training logs.
65b9c9b05b16ab2f991e5cd9
Yifan Peng
admin_assigned
2024-01-31T10:09:15.380Z
61809f31a367a8f5351ef353
https://cdn-avatars.huggingface.co/v1/production/uploads/61809f31a367a8f5351ef353/s5eQ00YeoirakzE_rJ0cy.jpeg
Yifan Peng
user
pyf98
65b9c9b05b16ab2f991e5cda
Jinchuan Tian
admin_assigned
2024-01-31T10:09:25.873Z
64770ff6d7cf39f2e937b882
/avatars/7b729501856f5073cf9e484811012f2a.svg
Jinchuan Tian
user
JinchuanTian
65b9c9b05b16ab2f991e5cdb
William Chen
claimed_verified
2024-06-25T07:34:35.865Z
630438615c70c21d0eae6613
https://cdn-avatars.huggingface.co/v1/production/uploads/630438615c70c21d0eae6613/KEHsB-znDIKHt7M0eyP8j.png
William Chen
user
wanchichen
65b9c9b05b16ab2f991e5cdc
Siddhant Arora
admin_assigned
2024-01-31T10:09:59.498Z
60d26d0d35274b8c1e4309f3
/avatars/aa0810324729268eb4a9aca78c95a7f0.svg
Arora
user
Siddhant
65b9c9b05b16ab2f991e5cdd
Brian Yan
admin_assigned
2024-01-31T10:10:07.423Z
620298a162b2b0e46e799698
/avatars/ae7ecde8b706e7add741100270d7cbc8.svg
Brian Yan
user
brianyan918
65b9c9b05b16ab2f991e5cde
Yui Sudo
admin_assigned
2024-01-31T10:10:14.428Z
6524aea9050781c16f324de5
/avatars/06eecfb2e779ef88bc3c39fc50591831.svg
Yui Sudo
user
yuis0510
65b9c9b05b16ab2f991e5cdf
Muhammad Shakeel
claimed_verified
2024-05-14T07:07:11.384Z
657821170766f274f3c9b35f
/avatars/3e95b1fb9578ec27d9ad176c34b979c4.svg
Muhammad Shakeel
user
shakeel1
65b9c9b05b16ab2f991e5ce0
Kwanghee Choi
admin_assigned
2024-01-31T10:11:08.812Z
64ba7c68710f5e5476877218
/avatars/1e5ad96dd0b237db1bfee400aa8ed229.svg
Kwanghee Choi
user
juice500
65b9c9b05b16ab2f991e5ce1
Jiatong Shi
admin_assigned
2024-01-31T10:11:16.422Z
6136ec7a40e43b8f748a0832
/avatars/089afa82ecfe6432d2dbfe81048c6ffa.svg
Jiatong Shi
user
ftshijt
65b9c9b05b16ab2f991e5ce2
Xuankai Chang
admin_assigned
2024-01-31T10:11:23.253Z
6136b5707c6d9b8e41469bec
/avatars/5b7f4ee0ba36bb6b87accda0ee27bb17.svg
Xuankai Chang
user
simpleoier
65b9c9b05b16ab2f991e5ce3
Jee-weon Jung
admin_assigned
2024-01-31T10:11:29.586Z
6226f40ee1cc4da2221557a0
/avatars/9654850f5aeba1d7c19ae017096c78d7.svg
Jee-weon Jung
user
jungjee
65b9c9b05b16ab2f991e5ce4
Shinji Watanabe
admin_assigned
2024-01-31T10:11:51.264Z
5feb9dbc29bcc7d2abb1cb76
/avatars/908d392634377e674ba9e5a4318b101e.svg
Shinji Watanabe
user
sw005320
OWSM v3.1: Better and Faster Open Whisper-Style Speech Models based on
  E-Branchformer
Recent studies have advocated for fully open foundation models to promote
transparency and open science. As an initial step, the Open Whisper-style
Speech Model (OWSM) reproduced OpenAI's Whisper using publicly available data
and open-source toolkits. With the aim of reproducing Whisper, the previous
OWSM v1 through v3 models were still based on Transformer, which might lead to
inferior performance compared to other state-of-the-art speech encoders. In
this work, we aim to improve the performance and efficiency of OWSM without
extra training data. We present E-Branchformer based OWSM v3.1 models at two
scales, i.e., 100M and 1B. The 1B model is the largest E-Branchformer based
speech model that has been made publicly available. It outperforms the previous
OWSM v3 in a vast majority of evaluation benchmarks, while demonstrating up to
25% faster inference speed. We publicly release the data preparation scripts,
pre-trained models and training logs.
E-Branchformer-based OWSM v3.1 models improve speech processing efficiency and performance without additional training data.
Transformer E-Branchformer speech encoders data preparation scripts pre-trained models training logs