When you use load_best_moel_at_end in combination with save_total_limit, the checkpoint with the best metric is never deleted (it’s always put first in the list of all checkpoints).
There is no way for now to keep the 5 best checkpoints, only the best one.