There are some options, none of which are completely satisfactory:

Compute a metric and not the loss during validation.
If memory is limited, you may have to reduce the batch size during validation, and not use the torch.no_grad() context manager, so you can compute the loss.