Limitations and Future Work

Data Scale Enhancing the scale of both data and
the model holds promising potential for yielding
significant improvements in quality. Following
(Dhariwal et al., 2020; Borsos et al., 2022), we
suggest training with 50K-100K hours instead of
2.5K. Computer Vision studies like Saharia et al.
(2022) show that utilizing larger pretrained lan-
guage models for text embeddings plays an im-
portant role in achieving better quality outcomes.
Drawing upon this, we hypothesize that the ap-
plication of a larger pretrained language model to
our second-stage model can similarly contribute to
enhanced quality outcomes.
MOUSAI