I don’t understand. My current structure is of the form:
Lots of ordinary modules from torch.nn->Conv2d->ReLU->ConvMask(my custom module)->ReLU->Conv2d->More ordinary modules from torch.nn
Shouldn’t the backpropagated gradients until the ReLU following the ConvMask already be cleaned of NaNs and Infs, since those are just ordinary modules?