9 Appendix
9.1 Model Release
As part of this work, we release the weights of the 20B checkpoint. The weights of the model can be found in
in this GCP bucket (gs://scenic-bucket/ul2). These checkpoints were trained with T5X (Roberts et al.,
2022) found at https://github.com/google-research/t5x and are implemented in JAX/Flax. Because the
ﬁne-tuning results are generally not from a single checkpoint due to our continuous ﬁnetuning setup, we
release three diﬀerent checkpoints (1.87M, 2.05M, 2.65M) which we found to be consistently pretty good.
A slight disclaimer is that we ﬁnetuned and trained this model on TPUv4 chips on our internal systems. Even
so, ﬁnetuning would also sometimes result in nans which may require some care and manual tuning to get
resolved. Therefore, if checkpoints were to be ported to another system, we could not guarantee that these
checkpoints would work as well. We are overall optimistic but we do not guarantee its stability with external
UL2- Unifying Language Learning Paradigms