text: Title: Very low-efficiency dataloader when no explicit matrix2numpy conversion using pykaldi+pytorch
  Body: Thanks to pykaldi. Now it is very easy to incorporate kaldi's feature into pytorch to do NN training related to speaker verification with only few lines of codes. 
  
  However, I found it is very slow to convert pykaldi's submatrix into pytorch's FloatTensor if there is no explicit conversion from submatrix to numpy. This leads the data loading phase become performance bottleneck when incorporating pykaldi into pytorch using its dataloader scheme.
  
  The initial problematic part of codes of my dataloader.py looks like this:
  ```python
  class Mydataset(object):
      ...
      def __getitem__(self, idx):
          uttid = self.uttids[idx]
          feat2d = utt2feat2d[uttid] # utt2feat2d is provided by read_utt2feat2d function
          label = labels[uttid]
          return torch.FloatTensor(feat2d), torch.LongTensor(label)
  
      def read_utt2feat2d(self, iopath2feat2d): # this function provides utt2featd
         rspec = 'scp:{0}/feat2d.scp'.format(iopath2feat2d)
         utt2feat2d = {}
        for key,val in SequentialMatrixReader(rspec):
             utt2feat2d[key] = val # Replacing it with utt2feat2d[key] = val.numpy() increases data loading speed
        return utt2feat2d.  # It occupies large amount of men, but the aim here is to debug
  ```
  
  The above codes result a slow dataloader. If the batch-size is 128, #workers 24,  it costs **4 mins** to load 150 batches.
  
  I checked the gpu (doing model training) consumes about 1mins, and the cpu (data loading) consumes
  4 mins. And they are nearly doing in parallel, the bottleneck lies in the data loading part.
  
  And If I simply revise the code of function read_utt2feat2d to:  **utt2feat2d[key] = val.numpy()**, the  total time consumed is **1min**. 
  
  
  I don't know the underlying reason and feel interested.