Thanks for your reply.
I want option #2.
However, it doesn’t work. I have written:
def score_function(engine):
        return evaluator.state.metrics["mse"]

handler = ModelCheckpoint(
        "models/", "checkpoint", score_function=score_function, n_saved=5
    )

trainer.add_event_handler(
        Events.EPOCH_COMPLETED(every=10), handler, {"textcnn": model}
    )

The problem is that it saves the first 5 models every 10 epochs, so the model at epoch 10,20,30,40,50 . However, it doesn’t then replace them as the training continues (although the mse does drop)…