Solved by ptrblck in post #2 
              
              
                You are not using the GPU, so a high CPU workload is expected. 
This tutorial explains how to use the GPU by pushing the model as well as the tensors to the device.