Thanks for following up with me.
First, the code takes up so much memory because I save result of every epoch into list and forget to release them like below:
class Network(nn.Module):
    def __init__(self):
        self.conv1 = nn.Conv2d()
        self.conv2 = nn.Conv2d()
        self.conv3 = nn.Conv2d()
        self.feature_list = []

    def forward(self, X):
        x = self.conv3(self.conv2(self.conv1(X)))
        self.feature_list.append(x)

       return self.feature_list

For dataparallel issue, as I suspected, my code has for loop for temporal/time axis, which I put in the first axis like [time, batch, channel, height, width]. So I have total 20 time-step with 4 gpu, DataParallel would split my input to 4 shares, each one with length 5. Thus when I try to index to 6th element, it report index out of range error.
Thanks a lot