I think the key thing is not whether opencv/numpy takes Variables. It is more because that what cv2.findcontours does is very likely a non-differentiable operation. You need to find differentiable alternative and write a custom loss. Maybe someone with more advanced vision background can help.