You are storing the batch_loss tensor with the entire computation graph in these lines of code:
        batch_loss = calc_loss(score, y_test)
        loss += batch_loss

which would thus also increase the memory usage.
You could avoid this by calculating the gradients inside the loop which would thus free the computation graph with its intermediate activation tensors.