I was trying to save the checkpoint after each epoch by setting load_best_model_at_end=True with p3.16xlarge parallel training. However, the following error occurs:
OSError
Can’t load config for ‘bert_model/checkpoint-156’. Make sure that:

‘bert_model/checkpoint-156’ is a correct model identifier listed on ‘Hugging Face – The AI community building the future. 1’
or ‘bert_model/checkpoint-156’ is the correct path to a directory containing config.json file.

The error occurs only if the output_dir folder is empty but not occurs if there were checkpoints from last training. Does anyone face the same issue or have an idea on this?