Forgot to include memory requirements…
Preferably will train on a dataset of input & output sequences with length 256, batch size 24, with a T5-base model.
T5-base has 220M params while OPTIMUS has 227M and was trained with the same params above using 8 v100GPUs.
The TPUv3-8 is equivalent to 4 v100 GPUs so should be able to train with at least batch size 12 or with shorter sequences.
OPTIMUS (current SOTA VAE) https://arxiv.org/pdf/2004.04092.pdf 3