Have you tried balancing the data in sk-learn prior to using it in PyTorch?
One example is provided here:

  
      nick becker – 23 Dec 16
  
  
    

The Right Way to Oversample in Predictive Modeling 56

Model Evaluation, Oversampling, Predictive Modeling


  
  
    
    
  
  


https://www.kaggle.com/varunsharma3/credit-card-fraud-detection-using-smote-0-99-auc 23