Have you followed the steps in this docs 120?
Could you post the custom method you are using, so that we could have a look? A simplified version would also be OK. Somehow the computation graph might be broken, and I assume the model works without this custom layer?