I found that the computation became slower and slower when training a huge volume of data with cuda. The batch size is 1003224*224. At the very beginning, it costs 0.4 second to train each chuck. Little by little, it costs nearly 3 seconds. In contrast, if selecting a small group of data from the original dataset for training, the speed keeps the same all along. Therefore, I wonder how to train a model on a large dataset. The main codes are simplified as follows to make the codes more readability.
x= torch.autograd.Variable(torch.zeros(batchSize,3,2*imgW,2*imgW)).cuda(0) 
yt=torch.autograd.Variable(torch.LongTensor(batchSize).zero_()).cuda(0)  
model = models.resnet18(pretrained=True).cuda(0) 
for epoch in range(0,epochs): 
    correct=0
        
    for t in range(0, trainSize, batchSize):  
        idx=0 
        for i in range(t, maxSize): # the batch of input x and target yt are assigned by local patches of # a large image here
             x[idx,:,:,:]=image[:, i-imgW:i+imgW, i-imgW:i+imgW] 
             yt[idx]=torch.from_numpy(Class[i]) 
             idx=idx+1    
        optimizer.zero_grad()   
        output = model(x) 
        loss = criterion(m(output), yt) 
        pred = output.data.max(1)[1] 
        correct += pred.eq(yt.data).sum()
        loss.backward()
        optimizer.step()

Thank you for your instruction beforehand.