I have figured out the issue, apparently the custom loss function I used was not working properly. PSA: Install pytorch from source, the current anaconda version has issues that prevent you from using BCELoss properly