I ran quick experiment running 4 data point in total and I get better performance if I am storing torch tensor rather than numpy. The experiment included running q network on 20 episodes in a deterministic environment
Torch: 2 min and 2 min 12 sec.
Numpy 3 min 34 sec. and 3 min 26 sec.
The only difference in the code includes converting the numpy array intor a torch array in order to run inference and learning process. I hope this helps.