Do you need to use the whole dataset for training? Is there any reason you couldn’t just train on 1 million rows? Or simply just train for 1-2 epochs on the whole dataset. One epoch would take about 16 days.
It looks like you are manually performing a linear activation function. Is there any reason you can’t use build in pytorch tools to do that?
One other thing that might be worth considering is decreasing your batch size and doing parallel data loading.