Are you storing the loss tensor in a list or another container?
This would increase the memory usage. However, the tensor should not be attached to the computation graph based on your description, which seems to be another unrelated issue.
Could you remove the new tensor creation and just try to use:
loss = loss_func(torch.round(inputs), targets) - 1

Also, I’m not sure about the derivative of round, but I assume it should be zeros almost everywhere?