Yeah that’s a tricky one. There’s no way to tell the DataLoader to dynamically skip some inputs and replace them with next ones, because it would violate the guarantee that it will always return the data in the same order as the given by the sampler.
Why is having variable batch size so bad? The only solution I can think of is to pre-process the whole dataset, and save a list of tuples: (filename, offset_x, offset_y) which would contain all non-zero patches. This would give you a sequence of all good patches for use at training time. You could add an option to preprocess it once and later pickle the data somewhere so it’s faster next time.