 Preprocessing  The texts are tokenized using a byte-level version of Byte Pair Encoding (BPE) (for unicode characters) and a vocabulary size of 50,257. The inputs are sequences of 1024 consecutive tokens.  The larger model was trained on 256 cloud TPU v3 cores. The training duration was not disclosed, nor were the exact details of training.  