Solved by albanD in post #2 
              
              
                Hi, 
This post was trying to do the same thing you do I think and the same answer applies to your code if I’m not mistaken: Make Custom Conv2d Layer efficient (wrt speed and memory)