How did you implement the communication between MXNet and PyTorch?
The GPU utilization might be low, if your workload is small or if your application has a bottleneck in another part of the code.
How large is your input and how fast is the data loading (including MXNet processing) compared to the forward pass of your PyTorch model?