Solved by Bhack in post #6 
              
              
                It is explained here in the doc: 

Since the function takes numpy arrays, you cannot take gradients through a numpy_function. If you require something that is differentiable, please consider using tf.py_function. 

If you just need a dilation you could try to add a dilation layer: 


We have also a …