In my case the dataset is extremely imbalanced (99.82% background and 3 other classes which make up for the remaining 0.18%), and I think this phenomenon takes place due to the inherent dataset imbalance present
Very curious though that UNET + specific encoder works on Windows with updated driver but on Linux fails.