For general performance improvements you could take a look at the Performance Guide 3. While you could potentially improve the performance I doubt it could explain the large difference.
Based on the model architecture, I would guess that the general overhead might be high compared to the actual GPU workload, which should be visible as whitespaces in a profiling timeline (i.e. no CUDA kernels being executed). To verify this, you could create an Nsight Systems 2 profile and check the timeline.