Well memory is finite. And cat is trying to put everything that you pass to it into one contiguous chunk. If self._make_layer is creating your SubtractedConv layers from above, then it seems to me it would be making (2 * filters) + 1 filters every time.
With all of that loaded into GPU memory, it is then trying to allocate enough space to concatenate your d and l tensors from above.
I’m not totally sure how cat works in allocating memory but a quick test seems to indicate that it is copying the concatenated tensors.

image.png1044×693 21.9 KB

Maybe you can try taking the concatenation out of your SubtractedConv, return the d and l from each layer and then cat a bunch of them all at once?