Add option to save on each training node
# What does this PR do?

There is currently a problem when using `load_best_model_at_end=True` on a training with multiple nodes: the model is only saved on the main process (so the machine rank 0) and machine with other ranks can't see the saved model (unless the system uses some kind of shared storage). This PR adds a flag to enable the save on each node for that situation, and avoids the hard failure when the model to reload is not found.