@pytorchnewbie score_function option is used to save only the best N models.
To resume, there are two options how to save models:

either based on any event. For example in like previous answer, model is always saved every 50-th epochs no matter its quality.
either based on a score. For example, evaluator engine provides a metric score, thus on the moment to save or not a model, Checkpoint verifies the current score, compares with registered score(s) and decides if we need or not to save current model.

If you would like combine both, you need to provide your own score_function that would take into account event number and your score.
trainer = ...
evaluator = ...

def score_function(_):
    # MAKE SURE THAT MSE SCORE IS AVAILABLE WHEN IT IS CALLED
    mse = evaluator.state.metrics["mse"]
    epoch = trainer.state.epoch
    # Create your own logic when to save the model
    # ... model with highest scores will be retained.
    return some_score

to_save = {'model': model}
handler = Checkpoint(..., score_function=score_function, ...)

trainer.add_event_handler(Events.COMPLETED, handler)

HTH
PS. If you want just to display eval score in the filename while saving a model every 50 epochs, unfortunately, this should be done by overriding the way you write the file…