Are you seeing the same error, if you pass all parameters to the optimizer?
If so, I guess you might be applying a differentiable operation on (some) parameters inside the CustomLayer, such as to().
Could you post the CustomLayer definition in this case, so that we could have a look?