 Training details  cT5 models used T5's weights as a starting point, and then it was finetuned on the  English [wikipedia](https://huggingface.co/datasets/wikipedia) for 3 epochs,  achieving ~74% validation accuracy (ct5-small). The training script is in JAX + Flax and can be found in `pretrain_ct5.py`.  Flax checkpoints can be converted to PyTorch via `convert_flax_to_pytorch.py [flax_dirname]`.   