Hey, I think you are mixing flax & pytorch scripts. how can you get pytorch error in flax script otherwise?
FlaxBigBird works perfectly on cloud TPU. You can refer this script 4 also.
Also Currently Trainer is for only PyTorch & Tensorflow.