Are your weights converging to 0 as well when using BCE?
If you’re using batch norm, what are the weights there?
I’d suspect your batch norm sigma would quickly move towards 0 to satisfy the unbalanced dataset. I.e. if you always predict non-tumor you already get a quite good loss (an obvious local minima).
Most people use focal loss nowadays instead of cross entropy. If you have a look at the original paper from retinanet (which introduces focal loss) they also use a scalar to normalise the known unbalance. E.g. you basically want to increase your tumor loss 9-fold to match the non-tumor loss. Does it make sense?