You can sum all training or test loss. something like this
test_loss = test_loss + loss.item()
After that, you can simply divide the total loss by the total number of batches per epoch.
alpha = (len(test_loader.dataset))/ batchsize
print(test_loss)
test_loss /= alpha