You are right. Sorry I misread the code.
I am running your original script on my box and seeing ~25% GPU usage constantly during training.
Here is a gist that shows how to add num_workers and pin_memory properly. Note that you need to wrap things in if __name__ == '__main__': for multiprocessing to work. It doubles my GPU usage. So you can try this as well.

  
      gist.github.com
  
  
    https://gist.github.com/SsnL/035fefea909325ef2a645a3398653585 177
cnn.py
import torch
import torch.nn as nn
import torchvision.datasets as dsets
import torchvision.transforms as transforms
from torch.autograd import Variable


# Hyper Parameters
num_epochs = 5
batch_size = 100
This file has been truncated. show original