If you store tensors, which are attached to the computation graph, you will also store the whole graph.
Usually this increases the memory so that you would hit an OOM issue after a while.
I don’t know, how these values are used in your code, but here are some potential condidates:
# in training_step
comet_logs = {'training_loss': loss}
return {'loss': loss, 'log': comet_logs}

# in validation step
output = OrderedDict({
    'val_loss': loss_val,
    "avg_cer": torch.tensor(avg_cer),
    "avg_wer": torch.tensor(avg_wer),
})

# in validation_epoch_end
comet_logs = OrderedDict({
    'val_loss': avg_loss,
    "avg_cer": avg_cer,
    "avg_wer": avg_wer}
)

If you are storing these dicts outside of your training and validation methods, this might explain the OOM. You could call .detach() on all these tensors, if you don’t need to call backward on them anymore to avoid this issue.
Also, you could wrap your validation loop in a with torch.no_grad() bock to save further memory.