You can try to extend the custom layer to process batches (all the layers in pytorch can process batches and most of the operations like matrix multiplication too)
and then process everything as a (b*c,h,w) tensor without envolving loops