Would crossentropy loss work if the target mask has shape Nx4xHxW? I thought that the mask for cross entropy needs to be NxHxW only.
I will try it.
I use empty_cache() because I actually thought that it does save up some gpu memory, the images are very large (and the batch size is very small, hence grad accumulation).