If you can run docker, you could try to run the code inside it.
Here are my results:
P100, 16GB
N = 1000
GPU Time: 0.076056s/epoch
N = 11000
GPU Time: 0.148172s/epoch
N = 21000
GPU Time: 0.220223s/epoch
N = 31000
GPU Time: 0.299151s/epoch
N = 41000
GPU Time: 0.368989s/epoch
N = 51000
GPU Time: 0.448879s/epoch

V100, 16GB
N = 1000
GPU Time: 0.081930s/epoch
N = 11000
GPU Time: 0.147144s/epoch
N = 21000
GPU Time: 0.254728s/epoch
N = 31000
GPU Time: 0.282772s/epoch
N = 41000
GPU Time: 0.382091s/epoch
N = 51000
GPU Time: 0.440113s/epoch

Note that I just executed the code once for each config.