You need to make sure that your custom layer does not break the flow of the gradients for its inputs.
In particular, you should not .detach() or unwrap/rewrap the tensor that is given as input (or use .data but that is a general rule everywhere).
Can you share the forward code for your custom_layer?