I was actually wrong to use BCELoss.
CrossEntropyLoss better suited my application and now provides an accuracy increase of 4% as well over both the validation and test sets with all kinds of batch sizes.
A lesson in understanding the right approach.