Hi,
I am running a Bert from scratch on google cloud, and it is working.
But when I am doing the training using a saved Checkpoint with “model_name_or_path” it makes the TPU crash with SIGKILL error (memory issue I guess).
I don’t understand this behavior, since if I rerun it without the checkpoint it works with no problems.
Running with checkpoint works only if I use just 1 core (num_cores=1) which is not convenient for me (takes a much larger time).
Anyone have an idea to help me?
Thanks