Thanks!
I still have some confusions when I followed this method.


What does batch mean
Suppose the csv data has (360000 - 200) lines, so I set
chunk_size = 3600
batch_size = 10
then I found the len(loader) is 10 (360000 = 3600 x 10 x 10) which indicate the batch is for chunks, but not for the lines in one chunk. Every batch it grabs 10 chunks, right?


In which place I should process data
The data processing procedure can be inserted into two places, the first place is in " getitem (self, index)", the other place is to handle the data in enumerate(loader), which place is best?


RuntimeError: inconsistent tensor sizes at THTensorMath.c
I think the data is just a batch of chunks(3600 x 10) in every enumerate(loader), maybe it’s ok to send 3600 x 10 lines to Net(lines), but before this step, it will meet a RuntimeError in enumerate(loader) place. The reason is the total lines is actually less than 360000, so there must be one loader which contains 9 batches of 3600 lines plus 1 batch which is less than 3600 lines(3600 - 200). I don’t how to fix this problem, should I expand every loader into lines?


The pseudo code is like this:
class myDataset(Dataset):
def init (self, path):
# init
def getitem (self, index):
data = read_chunck(dp)
label = read_chunck(lp)
return data, label
def __len__ (self):
    return math.ceil(filesize / chunksize)

…
.
.
data = myDataset()
loader = torch.utils.data.DataLoader(data, batch_size, shuffle=True)
for epoch in range(nepochs):
for i, (samples, labels) in enumerate(loader):    # RUNTIME ERROR
samples = Variable(samples)
labels = Variable(labels)
…
outputs = net(samples.float())
…
#!!!
return torch.cat(inputs, dim)
RuntimeError: inconsistent tensor sizes at /opt/conda/conda-bld/pytorch_1512386481460/work/torch/lib/TH/generic/THTensorMath.c:2864