I have seen a similar problem when using very small environments and NNs. In my case it was a 10 node(?) model with a single hidden layer.
In my case it was partially due to the swapping, but also my model was as small that it was not able to benefit from the parallel execution of the GPU, instead it suffered from the much lower clock rate.
So in case that your model is REALLY small this might be helpful