Correct me if I’m wrong, but what I have read in the official announcement is that we are going to get TPUv3-8 VM, which has 8 cores in total. Though I’m not entirely sure of its maximum capability to train a very large language model, other proposals have similarly proposed to pre-train language models from scratch as well, only in a different language. Let me know if I’m wrong though.
Cheers!