Solved by bPangolin in post #2 
              
              
                I was actually wrong to use BCELoss. 
CrossEntropyLoss better suited my application and now provides an accuracy increase of 4% as well over both the validation and test sets with all kinds of batch sizes. 
A lesson in understanding the right approach.