I have the same issue, so maybe his topic is also related to mine. I am handling data sets that describe certain chemicals on the ocean surface. You can see this as a map with valid values on the sea surface and a mask (or nans) on the land surface.
Ideally, I would like to ignore nans by using the same padding techniques as at the image borders and completely remove full nan fields after the flattening. Hereby, it is worth mentioning that the nans are at the same position for all images and features. Any hints on how I can do this?