Ok… For anyone that might land up here,  BCELoss seems to have an issue in PyTorch.  Switching to CrossEntropy loss even for a binary classification task, solved my problem.
In summary, if you architecture is right, double check the  choice of loss functions and the way the true labels have to be prepared, as expected by the loss function.