I was just following the tutorial here https://github.com/pytorch/pytorch/releases/tag/v0.4.0 37 and added py::call_guard<py::gil_scoped_release>() as the last argument to m.def in PYBIND11_MODULE.
That extension runs on GPU. It has some looping so it is CPU intensive as well.