The very large values are not causing memory problems for sure, but they might be the symptom of another issue.
I know I had issues when computing loss, if you have a tensor of size batch_size and another of size batch_size x 1 then because of the broadcasting semantic, if you sum or multiply element-wise these tensors, you will get a batch_size x batch_size tensor. This wrong size can easily be hidden when you average the loss with corr_loss = torhc.mean(some_tensor) for example. This kind of silent errors would cause both the surprisingly large loss and the unexpected memory usage.